Attachment 1480400 Details for Bug 1580024 – node logs

node logs

node-journal.log (text/x-vhdl), 8.30 MB, created by Alexis Solanas on 2018-09-02 22:39:23 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Alexis Solanas

Created: 2018-09-02 22:39:23 UTC

Size: 8.30 MB

patch

obsolete

>-- Logs begin at Fri 2018-08-31 04:30:27 EDT, end at Sun 2018-09-02 18:35:49 EDT. --
>Sep 02 00:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 00:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 00:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:00:37 Starting Node Health Status refresh
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:00:38 Cleaned 0 nodes from health cache
>Sep 02 00:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 46 of user root.
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[23415]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 46 of user root.
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[23418]: starting 0anacron
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[23424]: Anacron started on 2018-09-02
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[23424]: Normal exit (0 jobs run)
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[23426]: finished 0anacron
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 00:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 00:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:02:00.238708       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:02:37 Starting Node Health Status refresh
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:02:38 Cleaned 0 nodes from health cache
>Sep 02 00:02:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:02:57.826128   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:02:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:02:57.828860   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:03:09.123829   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:04:37 Starting Node Health Status refresh
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:04:38 Cleaned 0 nodes from health cache
>Sep 02 00:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:05:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:05:57.868683   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:05:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:05:57.869797   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:06:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:06:27.001827       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:06:37 Starting Node Health Status refresh
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:06:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:06:37 Cleaned 0 nodes from health cache
>Sep 02 00:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:11.415Z] [INFO] Attempting bootstrap at 2018-09-02 04:07:11.413673592 +0000 UTC
>Sep 02 00:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:13.336Z] [INFO] Validating specs...
>Sep 02 00:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:13.336Z] [NOTICE] All specs passed validation!
>Sep 02 00:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:07:13.756Z] [INFO] Validating specs...
>Sep 02 00:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:13.76Z] [NOTICE] All specs passed validation!
>Sep 02 00:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:07:13.972Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:08:09.127031   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:08:37 Starting Node Health Status refresh
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:08:38 Cleaned 0 nodes from health cache
>Sep 02 00:08:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:08:57.886357   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:08:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:08:57.887808   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:10:37 Starting Node Health Status refresh
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 18h ago
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 18h ago
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 18h ago
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:10:38 Cleaned 0 nodes from health cache
>Sep 02 00:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:11:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:11:09.313343       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:11:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:11:37.164314       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:11:50 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 45.33.48.4 replaced with 74.6.168.72
>Sep 02 00:11:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:11:57.932696   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:11:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:11:57.934080   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:12:37 Starting Node Health Status refresh
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:12:38 Cleaned 0 nodes from health cache
>Sep 02 00:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:13:09.128717   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:13:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:13:38.09Z] [INFO] AnsibleBroker::Catalog
>Sep 02 00:13:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:04:13:38 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 00:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:14:37 Starting Node Health Status refresh
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:14:38 Cleaned 0 nodes from health cache
>Sep 02 00:14:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:14:54.059707       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:14:58.022613   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:14:58.023823   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:16:37 Starting Node Health Status refresh
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:16:38 Cleaned 0 nodes from health cache
>Sep 02 00:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 04:17:11.413581344 +0000 UTC
>Sep 02 00:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:13.464Z] [INFO] Validating specs...
>Sep 02 00:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:13.467Z] [NOTICE] All specs passed validation!
>Sep 02 00:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:17:13.873Z] [INFO] Validating specs...
>Sep 02 00:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:13.873Z] [NOTICE] All specs passed validation!
>Sep 02 00:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:17:14.113Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:17:58.027215   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:17:58.028330   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:18:09.130302   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:18:37 Starting Node Health Status refresh
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:18:38 Cleaned 0 nodes from health cache
>Sep 02 00:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:19:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:19:39.343262       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:20:37 Starting Node Health Status refresh
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:20:38 Cleaned 0 nodes from health cache
>Sep 02 00:20:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:20:54.276385       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:20:58.092210   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:20:58.093776   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:22:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:22:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:22:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:22:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:22:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:22:37 Starting Node Health Status refresh
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:22:38 Cleaned 0 nodes from health cache
>Sep 02 00:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:23:09.131916   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:23:58.195146   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:23:58.196515   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:24:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:24:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:24:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:24:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:24:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:24:37 Starting Node Health Status refresh
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:24:38 Cleaned 0 nodes from health cache
>Sep 02 00:24:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:24:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:24:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:24:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:24:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:25:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:25:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:25:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:25:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:25:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:26:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:26:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:26:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:26:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:26:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:26:37 Starting Node Health Status refresh
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:26:38 Cleaned 0 nodes from health cache
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:26:58.240530   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:26:58.241652   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 04:27:11.413649104 +0000 UTC
>Sep 02 00:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:13.387Z] [INFO] Validating specs...
>Sep 02 00:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:13.388Z] [NOTICE] All specs passed validation!
>Sep 02 00:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:27:13.777Z] [INFO] Validating specs...
>Sep 02 00:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:13.778Z] [NOTICE] All specs passed validation!
>Sep 02 00:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:27:13.994Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:27:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:27:20.090906       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:28:09.133446   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:28:37 Starting Node Health Status refresh
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:28:38 Cleaned 0 nodes from health cache
>Sep 02 00:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:28:38.438271       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:29:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:29:22.313083       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:29:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:29:50.244Z] [INFO] AnsibleBroker::Catalog
>Sep 02 00:29:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:04:29:50 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:29:58.275858   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:29:58.276908   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:30:37 Starting Node Health Status refresh
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:30:38 Cleaned 0 nodes from health cache
>Sep 02 00:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:32:37 Starting Node Health Status refresh
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:32:38 Cleaned 0 nodes from health cache
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:32:58.332204   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:32:58.333305   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:33:09.135409   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:33:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:33:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:33:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:33:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:33:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:34:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:34:12.141015       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:34:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:34:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:34:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:34:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:34:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:34:37 Starting Node Health Status refresh
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:34:38 Cleaned 0 nodes from health cache
>Sep 02 00:34:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:34:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:34:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:34:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:34:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:35:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:35:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:35:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:35:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:35:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:35:58.338762   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:35:58.340294   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:35:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:36:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:36:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:36:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:36:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:36:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:36:37 Starting Node Health Status refresh
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:36:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:36:37 Cleaned 0 nodes from health cache
>Sep 02 00:36:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:36:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:36:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:36:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:36:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 04:37:11.413559665 +0000 UTC
>Sep 02 00:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:13.407Z] [INFO] Validating specs...
>Sep 02 00:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:13.407Z] [NOTICE] All specs passed validation!
>Sep 02 00:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:37:13.791Z] [INFO] Validating specs...
>Sep 02 00:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:13.791Z] [NOTICE] All specs passed validation!
>Sep 02 00:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:37:14.476Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:37:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:37:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:37:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:37:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:37:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:37:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:37:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:37:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:37:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:37:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:38:09.137759   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:38:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:38:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:38:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:38:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:38:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:38:37 Starting Node Health Status refresh
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:38:38 Cleaned 0 nodes from health cache
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:38:58.345575   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:38:58.346491   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:38:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:39:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:39:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:39:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:39:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:39:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:39:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:39:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:39:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:39:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:39:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:40:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:40:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:40:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:40:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:40:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:40:37 Starting Node Health Status refresh
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:40:38 Cleaned 0 nodes from health cache
>Sep 02 00:40:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:40:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:40:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:40:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:40:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:41:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:41:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:41:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:41:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:41:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:41:58.430635   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:41:58.431902   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:41:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:42:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:42:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:42:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:42:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:42:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:42:37 Starting Node Health Status refresh
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:42:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:42:37 Cleaned 0 nodes from health cache
>Sep 02 00:42:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:42:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:42:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:42:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:42:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:43:01.552873       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:43:09.139285   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:43:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:43:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:43:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:43:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:43:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:43:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:43:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:43:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:43:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:43:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:44:37 Starting Node Health Status refresh
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:44:38 Cleaned 0 nodes from health cache
>Sep 02 00:44:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:44:49.199981       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:44:58.440098   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:44:58.441307   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:44:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:45:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:45:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:45:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:45:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:45:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:45:34.394488       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:46:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:46:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:46:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:46:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:46:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:46:37 Starting Node Health Status refresh
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:46:38 Cleaned 0 nodes from health cache
>Sep 02 00:46:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:46:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:46:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:46:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:46:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 04:47:11.413575067 +0000 UTC
>Sep 02 00:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:13.49Z] [INFO] Validating specs...
>Sep 02 00:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:13.49Z] [NOTICE] All specs passed validation!
>Sep 02 00:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:47:13.898Z] [INFO] Validating specs...
>Sep 02 00:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:13.898Z] [NOTICE] All specs passed validation!
>Sep 02 00:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:47:14.029Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:47:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:47:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:47:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:47:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:47:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:47:58.458393   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:47:58.459442   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:48:09.140730   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:48:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:48:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:48:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:48:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:48:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:48:37 Starting Node Health Status refresh
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:48:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:48:37 Cleaned 0 nodes from health cache
>Sep 02 00:48:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:48:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:48:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:48:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:48:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:49:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:49:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:49:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:49:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:49:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:49:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:49:50.243Z] [INFO] AnsibleBroker::Catalog
>Sep 02 00:49:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:04:49:50 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 00:49:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:49:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:49:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:49:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:49:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:50:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:50:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:50:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:50:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:50:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:50:37 Starting Node Health Status refresh
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:50:38 Cleaned 0 nodes from health cache
>Sep 02 00:50:47 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 23.92.29.245 replaced with 45.32.199.189
>Sep 02 00:50:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 04:50:57.592927       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:50:58.513607   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:50:58.514630   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:51:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:51:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:51:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:51:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:51:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:51:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:51:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:51:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:51:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:51:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:52:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:52:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:52:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:52:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:52:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:52:37 Starting Node Health Status refresh
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:52:38 Cleaned 0 nodes from health cache
>Sep 02 00:52:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:52:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:52:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:52:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:52:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:53:09.142047   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:53:58.542369   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:53:58.543530   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:53:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:54:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:54:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:54:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:54:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:54:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:54:37 Starting Node Health Status refresh
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:54:38 Cleaned 0 nodes from health cache
>Sep 02 00:54:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:54:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:54:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:54:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:54:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:55:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:55:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:55:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:55:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:55:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:55:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:55:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:55:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:55:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:55:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:56:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:56:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:56:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:56:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:56:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:56:37 Starting Node Health Status refresh
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:56:38 Cleaned 0 nodes from health cache
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:56:58.577502   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:56:58.578662   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:56:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 00:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 04:57:11.413578204 +0000 UTC
>Sep 02 00:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 00:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:13.416Z] [INFO] Validating specs...
>Sep 02 00:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:13.417Z] [NOTICE] All specs passed validation!
>Sep 02 00:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T04:57:13.822Z] [INFO] Validating specs...
>Sep 02 00:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:13.822Z] [NOTICE] All specs passed validation!
>Sep 02 00:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T04:57:14.162Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 00:57:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:57:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:57:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:57:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:57:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:57:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:57:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:57:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:57:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:57:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 00:58:09.143963   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 00:58:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:58:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:58:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:58:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:58:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:58:37 Starting Node Health Status refresh
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 00:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 04:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 04:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 00:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 04:58:38 Cleaned 0 nodes from health cache
>Sep 02 00:58:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:58:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:58:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:58:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:58:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:59:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:59:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:59:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:59:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:59:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:59:58.583640   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 00:59:58.584780   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 00:59:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:00:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:00:16.289695       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 01:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 01:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:00:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:00:37 Starting Node Health Status refresh
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:00:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:00:37 Cleaned 0 nodes from health cache
>Sep 02 01:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:00:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 47 of user root.
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 47 of user root.
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[29310]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[29313]: starting 0anacron
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[29319]: Anacron started on 2018-09-02
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[29319]: Normal exit (0 jobs run)
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[29321]: finished 0anacron
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 01:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 01:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:01:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:01:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:01:54.934118       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:01:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:02:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:02:29.660766   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:02:29.665633   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:02:37 Starting Node Health Status refresh
>Sep 02 01:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:02:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:02:41 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:02:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:02:41 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:02:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:02:47 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:02:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:02:47 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:02:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:02:49.660635   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:02:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:02:49.665897   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:02:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:02:50 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:02:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:02:50 Cleaned 0 nodes from health cache
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:02:58.589520   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:02:58.590588   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:02:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:03:09.147020   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:03:09.660725   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:03:09.665754   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:03:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:03:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:04:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:04:37 Starting Node Health Status refresh
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:04:38 Cleaned 0 nodes from health cache
>Sep 02 01:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:04:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:05:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:05:58.681900   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:05:58.683072   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:06:02.704379       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:06:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:06:37 Starting Node Health Status refresh
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:06:38 Cleaned 0 nodes from health cache
>Sep 02 01:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:06:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:07:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:07.099Z] [INFO] AnsibleBroker::Catalog
>Sep 02 01:07:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:05:07:07 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 01:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 05:07:11.413572596 +0000 UTC
>Sep 02 01:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:13.464Z] [INFO] Validating specs...
>Sep 02 01:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:13.465Z] [NOTICE] All specs passed validation!
>Sep 02 01:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:07:13.863Z] [INFO] Validating specs...
>Sep 02 01:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:13.863Z] [NOTICE] All specs passed validation!
>Sep 02 01:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:07:14.428Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:07:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:07:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:08:09.148991   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:08:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:08:37 Starting Node Health Status refresh
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:08:38 Cleaned 0 nodes from health cache
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:08:58.727673   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:08:58.728612   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:08:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:09:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:09:32.316307       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:10:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:10:37 Starting Node Health Status refresh
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 19h ago
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 19h ago
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 19h ago
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:10:38 Cleaned 0 nodes from health cache
>Sep 02 01:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:10:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:11:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:11:58.785355   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:11:58.786556   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:11:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:12:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:12:37 Starting Node Health Status refresh
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:12:38 Cleaned 0 nodes from health cache
>Sep 02 01:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:12:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:13:09.151678   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:13:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:13:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:13:59.651974       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:14:37 Starting Node Health Status refresh
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:14:38 Cleaned 0 nodes from health cache
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:14:58.865947   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:14:58.867357   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:14:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:15:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:15:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:15:39.721283       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:16:37 Starting Node Health Status refresh
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:16:38 Cleaned 0 nodes from health cache
>Sep 02 01:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:16:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 05:17:11.413640941 +0000 UTC
>Sep 02 01:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:14.652Z] [INFO] Validating specs...
>Sep 02 01:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:14.653Z] [NOTICE] All specs passed validation!
>Sep 02 01:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:17:15.054Z] [INFO] Validating specs...
>Sep 02 01:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:15.055Z] [NOTICE] All specs passed validation!
>Sep 02 01:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:17:15.258Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:17:58.952189   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:17:58.953532   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:17:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:18:09.153388   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:18:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:18:37 Starting Node Health Status refresh
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:18:38 Cleaned 0 nodes from health cache
>Sep 02 01:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:18:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:20:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:20:37 Starting Node Health Status refresh
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:20:38 Cleaned 0 nodes from health cache
>Sep 02 01:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:20:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:20:59.021514   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:20:59.022389   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:21:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:21:47.411741       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:22:37 Starting Node Health Status refresh
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:22:38 Cleaned 0 nodes from health cache
>Sep 02 01:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:23:09.154566   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:23:55 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 66.70.234.51 replaced with 64.113.44.54
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:23:59.025772   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:23:59.027060   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:24:37 Starting Node Health Status refresh
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:24:38 Cleaned 0 nodes from health cache
>Sep 02 01:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:26:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:26:20.234Z] [INFO] AnsibleBroker::Catalog
>Sep 02 01:26:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:05:26:20 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 01:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:26:37 Starting Node Health Status refresh
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:26:38 Cleaned 0 nodes from health cache
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:26:59.122763   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:26:59.123979   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 05:27:11.413665096 +0000 UTC
>Sep 02 01:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:13.161Z] [INFO] Validating specs...
>Sep 02 01:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:13.163Z] [NOTICE] All specs passed validation!
>Sep 02 01:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:27:13.583Z] [INFO] Validating specs...
>Sep 02 01:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:13.584Z] [NOTICE] All specs passed validation!
>Sep 02 01:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:27:13.828Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:28:09.155931   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:28:37 Starting Node Health Status refresh
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:28:38 Cleaned 0 nodes from health cache
>Sep 02 01:28:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:28:50.827691       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:29:59.133971   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:29:59.134815   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:30:37 Starting Node Health Status refresh
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:30:38 Cleaned 0 nodes from health cache
>Sep 02 01:30:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:30:44.139466       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:32:37 Starting Node Health Status refresh
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:32:38 Cleaned 0 nodes from health cache
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:32:59.228630   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:32:59.229832   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:33:09.157946   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:34:37 Starting Node Health Status refresh
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:34:38 Cleaned 0 nodes from health cache
>Sep 02 01:34:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:34:54.497665       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:35:59.289927   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:35:59.291415   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:36:37 Starting Node Health Status refresh
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:36:38 Cleaned 0 nodes from health cache
>Sep 02 01:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 05:37:11.413581483 +0000 UTC
>Sep 02 01:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:13.512Z] [INFO] Validating specs...
>Sep 02 01:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:13.513Z] [NOTICE] All specs passed validation!
>Sep 02 01:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:37:13.919Z] [INFO] Validating specs...
>Sep 02 01:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:13.919Z] [NOTICE] All specs passed validation!
>Sep 02 01:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:37:14.047Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:38:09.161345   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:38:37 Starting Node Health Status refresh
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:38:38 Cleaned 0 nodes from health cache
>Sep 02 01:38:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:38:39.860176       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:38:59.346669   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:38:59.347866   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:40:37 Starting Node Health Status refresh
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:40:38 Cleaned 0 nodes from health cache
>Sep 02 01:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:41:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:41:57.285Z] [INFO] AnsibleBroker::Catalog
>Sep 02 01:41:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:05:41:57 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:41:59.409925   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:41:59.411035   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:42:37 Starting Node Health Status refresh
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:42:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:42:37 Cleaned 0 nodes from health cache
>Sep 02 01:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:43:09.162772   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:44:37 Starting Node Health Status refresh
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:44:38 Cleaned 0 nodes from health cache
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:44:59.505598   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:44:59.506485   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:45:34.232188       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:46:37 Starting Node Health Status refresh
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:46:38 Cleaned 0 nodes from health cache
>Sep 02 01:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 05:47:11.413580364 +0000 UTC
>Sep 02 01:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:13.771Z] [INFO] Validating specs...
>Sep 02 01:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:13.771Z] [NOTICE] All specs passed validation!
>Sep 02 01:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:47:14.193Z] [INFO] Validating specs...
>Sep 02 01:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:14.193Z] [NOTICE] All specs passed validation!
>Sep 02 01:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:47:14.48Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:47:59.534981   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:47:59.536108   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:48:09.164320   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:48:37 Starting Node Health Status refresh
>Sep 02 01:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:48:38 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:48:38 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:48:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:48:39 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:48:39 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:48:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:48:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:48:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:48:39 Cleaned 0 nodes from health cache
>Sep 02 01:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:50:37 Starting Node Health Status refresh
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:50:38 Cleaned 0 nodes from health cache
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:50:59.634934   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:50:59.635882   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:51:05.540930       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:52:37 Starting Node Health Status refresh
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:52:37.945094       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:52:38 Cleaned 0 nodes from health cache
>Sep 02 01:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:53:09.165648   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:53:59.671569   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:53:59.672434   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:54:37 Starting Node Health Status refresh
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:54:38 Cleaned 0 nodes from health cache
>Sep 02 01:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:56:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:56:09.660895   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:56:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:56:09.666549   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:56:29.660668   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:56:29.665590   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:56:37 Starting Node Health Status refresh
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:56:38 Cleaned 0 nodes from health cache
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:56:59.689948   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:56:59.690862   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 01:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 05:57:11.414631699 +0000 UTC
>Sep 02 01:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 01:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:14.96Z] [INFO] Validating specs...
>Sep 02 01:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:14.961Z] [NOTICE] All specs passed validation!
>Sep 02 01:57:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T05:57:15.355Z] [INFO] Validating specs...
>Sep 02 01:57:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:15.355Z] [NOTICE] All specs passed validation!
>Sep 02 01:57:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T05:57:15.739Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 01:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 01:58:09.167384   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 01:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:58:37 Starting Node Health Status refresh
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 01:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 05:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 05:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 01:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 05:58:38 Cleaned 0 nodes from health cache
>Sep 02 01:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:59:06.030906       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 05:59:30.330730       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:59:59.740159   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 01:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 01:59:59.741334   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:00:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:00:11.449Z] [INFO] AnsibleBroker::Catalog
>Sep 02 02:00:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:06:00:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 02:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 02:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 02:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:00:37 Starting Node Health Status refresh
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:00:38 Cleaned 0 nodes from health cache
>Sep 02 02:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 48 of user root.
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 48 of user root.
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[2741]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[2744]: starting 0anacron
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[2750]: Anacron started on 2018-09-02
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[2750]: Normal exit (0 jobs run)
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[2752]: finished 0anacron
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 02:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 02:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:02:37 Starting Node Health Status refresh
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:02:38 Cleaned 0 nodes from health cache
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:02:59.839884   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:02:59.842741   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:03:09.168685   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:04:37 Starting Node Health Status refresh
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:04:38 Cleaned 0 nodes from health cache
>Sep 02 02:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:05:59.925705   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:05:59.926908   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:06:03.813791       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:06:25 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 108.61.73.243 replaced with 50.116.52.97
>Sep 02 02:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:06:37 Starting Node Health Status refresh
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:06:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:06:37 Cleaned 0 nodes from health cache
>Sep 02 02:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:11.415Z] [INFO] Attempting bootstrap at 2018-09-02 06:07:11.413558405 +0000 UTC
>Sep 02 02:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:13.907Z] [INFO] Validating specs...
>Sep 02 02:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:13.909Z] [NOTICE] All specs passed validation!
>Sep 02 02:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:07:14.29Z] [INFO] Validating specs...
>Sep 02 02:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:14.291Z] [NOTICE] All specs passed validation!
>Sep 02 02:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:07:14.58Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:08:09.170002   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:08:37 Starting Node Health Status refresh
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:08:38 Cleaned 0 nodes from health cache
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:08:59.928726   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:08:59.929829   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:10:37 Starting Node Health Status refresh
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 20h ago
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 20h ago
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 20h ago
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:10:38 Cleaned 0 nodes from health cache
>Sep 02 02:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:11:59.948138   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:11:59.949316   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:12:37 Starting Node Health Status refresh
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:12:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:12:37 Cleaned 0 nodes from health cache
>Sep 02 02:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:13:09.171562   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:14:21.357899       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:14:37 Starting Node Health Status refresh
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:14:38 Cleaned 0 nodes from health cache
>Sep 02 02:14:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:14:40.090152       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:14:59.958339   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:14:59.959807   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:15:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:15:17.870359       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:15:58.584Z] [INFO] AnsibleBroker::Catalog
>Sep 02 02:15:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:06:15:58 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 02:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:16:37 Starting Node Health Status refresh
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:16:38 Cleaned 0 nodes from health cache
>Sep 02 02:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 06:17:11.413537968 +0000 UTC
>Sep 02 02:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:13.58Z] [INFO] Validating specs...
>Sep 02 02:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:13.58Z] [NOTICE] All specs passed validation!
>Sep 02 02:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:17:14.011Z] [INFO] Validating specs...
>Sep 02 02:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:14.011Z] [NOTICE] All specs passed validation!
>Sep 02 02:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:17:14.21Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:18:00.020771   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:18:00.021723   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:18:09.172935   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:18:37 Starting Node Health Status refresh
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:18:38 Cleaned 0 nodes from health cache
>Sep 02 02:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:20:37 Starting Node Health Status refresh
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:20:38 Cleaned 0 nodes from health cache
>Sep 02 02:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:21:00.079980   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:21:00.080974   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:22:37 Starting Node Health Status refresh
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:22:38 Cleaned 0 nodes from health cache
>Sep 02 02:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:23:09.174263   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:24:00.137743   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:24:00.139056   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:24:05.972405       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:24:37 Starting Node Health Status refresh
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:24:38 Cleaned 0 nodes from health cache
>Sep 02 02:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:24:38.126890       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:26:29.660874   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:26:29.666516   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:26:37 Starting Node Health Status refresh
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:26:38 Cleaned 0 nodes from health cache
>Sep 02 02:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:27:00.163154   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:27:00.164319   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 06:27:11.413620596 +0000 UTC
>Sep 02 02:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:13.213Z] [INFO] Validating specs...
>Sep 02 02:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:13.214Z] [NOTICE] All specs passed validation!
>Sep 02 02:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:27:13.611Z] [INFO] Validating specs...
>Sep 02 02:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:13.611Z] [NOTICE] All specs passed validation!
>Sep 02 02:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:27:13.836Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:28:09.175893   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:28:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:28:37 Starting Node Health Status refresh
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:28:38 Cleaned 0 nodes from health cache
>Sep 02 02:28:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:28:44.471411       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:30:00.210059   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:30:00.211114   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:30:37 Starting Node Health Status refresh
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:30:38 Cleaned 0 nodes from health cache
>Sep 02 02:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:32:37 Starting Node Health Status refresh
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:32:38 Cleaned 0 nodes from health cache
>Sep 02 02:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:33:00.289673   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:33:00.290684   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:33:09.177144   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:34:00.064912       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:34:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:34:14.665Z] [INFO] AnsibleBroker::Catalog
>Sep 02 02:34:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:06:34:14 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 02:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:34:37 Starting Node Health Status refresh
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:34:38 Cleaned 0 nodes from health cache
>Sep 02 02:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:35:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:36:00.380685   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:36:00.381699   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:36:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:36:37 Starting Node Health Status refresh
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:36:38 Cleaned 0 nodes from health cache
>Sep 02 02:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:36:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 06:37:11.413572745 +0000 UTC
>Sep 02 02:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:13.418Z] [INFO] Validating specs...
>Sep 02 02:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:13.42Z] [NOTICE] All specs passed validation!
>Sep 02 02:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:37:13.829Z] [INFO] Validating specs...
>Sep 02 02:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:13.829Z] [NOTICE] All specs passed validation!
>Sep 02 02:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:37:13.981Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:37:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:37:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:38:09.178426   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:38:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:38:37 Starting Node Health Status refresh
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:38:38 Cleaned 0 nodes from health cache
>Sep 02 02:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:38:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:39:00.478336   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:39:00.479317   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:39:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:39:07.251770       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: DHCPREQUEST on eth0 to 10.10.94.229 port 67 (xid=0x5127a64f)
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: DHCPACK from 10.10.94.229 (xid=0x5127a64f)
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9309] dhcp4 (eth0):   address 10.10.94.27
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9324] dhcp4 (eth0):   plen 22 (255.255.252.0)
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9329] dhcp4 (eth0):   classless static route 169.254.169.254/32 gw 10.10.94.229
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9333] dhcp4 (eth0):   gateway 10.10.95.254
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9338] dhcp4 (eth0):   lease time 86400
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9343] dhcp4 (eth0):   hostname 'host-10-10-94-27'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9347] dhcp4 (eth0):   nameserver '10.11.5.4'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9347] dhcp4 (eth0):   nameserver '10.11.5.3'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9347] dhcp4 (eth0):   domain name 'openstacklocal'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535870362.9347] dhcp4 (eth0): state changed bound -> bound
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dbus[470]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Network Manager Script Dispatcher Service...
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: bound to 10.10.94.27 -- renewal in 39324 seconds.
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dbus[470]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Network Manager Script Dispatcher Service.
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: req:1 'dhcp4-change' [eth0]: new request (6 scripts)
>Sep 02 02:39:22 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + cd /etc/sysconfig/network-scripts
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + . ./network-functions
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ export PATH
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ hostname
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ HOSTNAME=node-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ . /etc/init.d/functions
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ TEXTDOMAIN=initscripts
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ umask 022
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ export PATH
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' 6534 -ne 1 -a -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -d /run/systemd/system ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ case "$0" in
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ COLUMNS=80
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -c /dev/stderr -a -r /dev/stderr ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ CONSOLETYPE=serial
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ . /etc/profile.d/lang.sh
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ unset LANGSH_SOURCED
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -z '' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' -f /etc/sysconfig/init ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ . /etc/sysconfig/init
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ BOOTUP=color
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ RES_COL=60
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ MOVE_TO_COL='echo -en \033[60G'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ SETCOLOR_FAILURE='echo -en \033[0;31m'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ SETCOLOR_WARNING='echo -en \033[0;33m'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ SETCOLOR_NORMAL='echo -en \033[0;39m'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' serial = serial ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ BOOTUP=serial
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ MOVE_TO_COL=
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ SETCOLOR_SUCCESS=
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ SETCOLOR_FAILURE=
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ SETCOLOR_WARNING=
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ SETCOLOR_NORMAL=
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ __sed_discard_ignored_files='/$~\|\.bak\|\.old\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave$$/d'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' '' = 1 ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++++ cat /proc/cmdline
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ strstr 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' rc.debug
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ '[' 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' = 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ return 1
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: +++ return 0
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + '[' -f ../network ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + . ../network
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ NETWORKING=yes
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ NOZEROCONF=yes
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + [[ dhcp4-change =~ ^(up|dhcp4-change|dhcp6-change)$ ]]
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + NEEDS_RESTART=0
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + UPSTREAM_DNS=/etc/dnsmasq.d/origin-upstream-dns.conf
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ mktemp
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + UPSTREAM_DNS_TMP=/tmp/tmp.tBDFG4QTW3
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ mktemp
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + UPSTREAM_DNS_TMP_SORTED=/tmp/tmp.TOBNDw1uOg
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ mktemp
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + CURRENT_UPSTREAM_DNS_SORTED=/tmp/tmp.6ND0hvkGa3
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ mktemp
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + NEW_RESOLV_CONF=/tmp/tmp.9lxb6NKlkr
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ mktemp
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + NEW_NODE_RESOLV_CONF=/tmp/tmp.wpdnhWBxA8
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ awk '{print $3 }'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ /sbin/ip route list match 0.0.0.0/0
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + def_route=10.10.95.254
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ awk '{print $3}'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ /sbin/ip route get to 10.10.95.254
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + def_route_int=eth0
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ awk '{print $5}'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ /sbin/ip route get to 10.10.95.254
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + def_route_ip=10.10.94.27
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + [[ eth0 == eth0 ]]
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + '[' '!' -f /etc/dnsmasq.d/origin-dns.conf ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + grep -q 99-origin-dns.sh /etc/resolv.conf
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ systemctl -q is-active dnsmasq.service
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + '[' 0 -eq 1 ']'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: ++ systemctl -q is-active dnsmasq.service
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + grep -q 99-origin-dns.sh /etc/resolv.conf
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + sed -e '/^nameserver.*$/d' /etc/resolv.conf
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + echo 'nameserver 10.10.94.27'
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + grep -qw search /tmp/tmp.9lxb6NKlkr
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + grep -q 'search cluster.local' /tmp/tmp.9lxb6NKlkr
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + cp -Z /tmp/tmp.9lxb6NKlkr /etc/resolv.conf
>Sep 02 02:39:23 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6534]: + rm -f /tmp/tmp.tBDFG4QTW3 /tmp/tmp.TOBNDw1uOg /tmp/tmp.6ND0hvkGa3 /tmp/tmp.9lxb6NKlkr
>Sep 02 02:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:39:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:39:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:40:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:40:37 Starting Node Health Status refresh
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:40:38 Cleaned 0 nodes from health cache
>Sep 02 02:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:40:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:41:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:41:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:42:00.485755   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:42:00.488633   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:42:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:42:24.590832       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:42:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:42:37 Starting Node Health Status refresh
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:42:38 Cleaned 0 nodes from health cache
>Sep 02 02:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:42:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:43:09.181699   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:43:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:43:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:44:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:44:37 Starting Node Health Status refresh
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:44:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:44:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:44:38 Cleaned 0 nodes from health cache
>Sep 02 02:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:44:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:45:00.513443   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:45:00.514411   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:45:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:45:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:46:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:46:37 Starting Node Health Status refresh
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:46:38 Cleaned 0 nodes from health cache
>Sep 02 02:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 06:47:11.41366296 +0000 UTC
>Sep 02 02:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:13.39Z] [INFO] Validating specs...
>Sep 02 02:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:13.391Z] [NOTICE] All specs passed validation!
>Sep 02 02:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:47:13.782Z] [INFO] Validating specs...
>Sep 02 02:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:13.782Z] [NOTICE] All specs passed validation!
>Sep 02 02:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:47:14.046Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:47:20 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 74.6.168.72 replaced with 199.102.46.77
>Sep 02 02:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:47:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:47:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:48:00.607547   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:48:00.608923   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:48:09.184187   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:48:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:48:37 Starting Node Health Status refresh
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:48:38 Cleaned 0 nodes from health cache
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:48:59.137911       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:48:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:49:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:49:26.688017       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:49:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:50:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:50:37 Starting Node Health Status refresh
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:50:38 Cleaned 0 nodes from health cache
>Sep 02 02:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:50:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:51:00.642504   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:51:00.643648   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:51:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:51:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:52:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:52:37 Starting Node Health Status refresh
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:52:38 Cleaned 0 nodes from health cache
>Sep 02 02:52:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:52:49.660769   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:52:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:52:49.665532   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:52:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:53:09.185770   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:53:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:54:00.715093   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:54:00.716012   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:54:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:54:09.884Z] [INFO] AnsibleBroker::Catalog
>Sep 02 02:54:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:06:54:09 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 02:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:54:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:54:37 Starting Node Health Status refresh
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:54:38 Cleaned 0 nodes from health cache
>Sep 02 02:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:55:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:55:59.660873   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:55:59.665578   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:55:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:56:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 06:56:32.320559       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 02:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:56:37 Starting Node Health Status refresh
>Sep 02 02:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:56:39 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:56:39 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:56:39 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:56:39 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:56:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:56:39 Cleaned 0 nodes from health cache
>Sep 02 02:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:57:00.788464   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 02:57:00.789307   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 02:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 02:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 06:57:11.413707987 +0000 UTC
>Sep 02 02:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 02:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 2018-09-02 06:57:13.378584 W | wal: sync duration of 1.712940653s, expected less than 1s
>Sep 02 02:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:13.555Z] [INFO] Validating specs...
>Sep 02 02:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:13.555Z] [NOTICE] All specs passed validation!
>Sep 02 02:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T06:57:13.969Z] [INFO] Validating specs...
>Sep 02 02:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:13.971Z] [NOTICE] All specs passed validation!
>Sep 02 02:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T06:57:14.139Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 02:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:57:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 02:58:09.189981   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 02:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:58:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:58:37 Starting Node Health Status refresh
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 02:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 06:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 06:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 02:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 06:58:38 Cleaned 0 nodes from health cache
>Sep 02 02:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:58:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:59:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 02:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 02:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 02:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 02:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 02:59:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:00:00.802386   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:00:00.803519   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 03:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 03:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:00:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:00:37 Starting Node Health Status refresh
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:00:38 Cleaned 0 nodes from health cache
>Sep 02 03:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 49 of user root.
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 49 of user root.
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[8696]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[8699]: starting 0anacron
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Anacron started on 2018-09-02
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Will run job `cron.daily' in 37 min.
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Jobs will be executed sequentially
>Sep 02 03:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[8707]: finished 0anacron
>Sep 02 03:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:01:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:01:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:02:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:02:37 Starting Node Health Status refresh
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:02:38 Cleaned 0 nodes from health cache
>Sep 02 03:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:02:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:03:00.881442   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:03:00.883751   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:03:09.192683   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:03:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:03:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:04:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:04:37 Starting Node Health Status refresh
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:04:38 Cleaned 0 nodes from health cache
>Sep 02 03:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:04:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:05:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:05:10.191735       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:05:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:05:13.800120       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:05:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:05:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:05:58.396638       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:05:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:06:00.948218   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:06:00.949305   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:06:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:06:37 Starting Node Health Status refresh
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:06:38 Cleaned 0 nodes from health cache
>Sep 02 03:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:06:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 07:07:11.413601877 +0000 UTC
>Sep 02 03:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:13.547Z] [INFO] Validating specs...
>Sep 02 03:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:13.549Z] [NOTICE] All specs passed validation!
>Sep 02 03:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:07:13.96Z] [INFO] Validating specs...
>Sep 02 03:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:13.962Z] [NOTICE] All specs passed validation!
>Sep 02 03:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:07:14.291Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:07:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:07:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:08:09.194523   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:08:37 Starting Node Health Status refresh
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:08:38 Cleaned 0 nodes from health cache
>Sep 02 03:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:08:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:09:00.997925   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:09:00.999157   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:09:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:09:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:10:37 Starting Node Health Status refresh
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 21h ago
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 21h ago
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 21h ago
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:10:38 Cleaned 0 nodes from health cache
>Sep 02 03:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:10:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:11:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:11:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:12:01.099662   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:12:01.100611   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:12:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:12:37 Starting Node Health Status refresh
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:12:38 Cleaned 0 nodes from health cache
>Sep 02 03:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:12:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:13:09.195966   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:13:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:13:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:13:55.773Z] [INFO] AnsibleBroker::Catalog
>Sep 02 03:13:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:07:13:55 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:13:59.499749       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:13:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:14:10 node-0.alxcns2.lab.rdu2.cee.redhat.com rhnsd[908]: /etc/sysconfig/rhn/systemid does not exist or is unreadable
>Sep 02 03:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:14:37 Starting Node Health Status refresh
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:14:38 Cleaned 0 nodes from health cache
>Sep 02 03:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:14:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:15:01.108399   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:15:01.109299   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:15:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:15:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:16:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:16:37 Starting Node Health Status refresh
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:16:38 Cleaned 0 nodes from health cache
>Sep 02 03:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:16:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 07:17:11.41444687 +0000 UTC
>Sep 02 03:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:13.853Z] [INFO] Validating specs...
>Sep 02 03:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:13.853Z] [NOTICE] All specs passed validation!
>Sep 02 03:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:17:14.244Z] [INFO] Validating specs...
>Sep 02 03:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:14.246Z] [NOTICE] All specs passed validation!
>Sep 02 03:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:17:14.605Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:17:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:17:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:18:01.155690   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:18:01.156824   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:18:09.198748   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:18:37 Starting Node Health Status refresh
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:18:38 Cleaned 0 nodes from health cache
>Sep 02 03:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:18:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:19:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:19:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:19:56.303659       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:19:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:20:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:20:37 Starting Node Health Status refresh
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:20:38 Cleaned 0 nodes from health cache
>Sep 02 03:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:20:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:21:01.227974   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:21:01.229076   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:21:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:21:59.947763       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:22:37 Starting Node Health Status refresh
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:22:38 Cleaned 0 nodes from health cache
>Sep 02 03:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:23:09.200368   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:23:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:24:01.289726   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:24:01.290612   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:24:37 Starting Node Health Status refresh
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:24:38 Cleaned 0 nodes from health cache
>Sep 02 03:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:25:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:25:44 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 45.32.199.189 replaced with 45.63.111.145
>Sep 02 03:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:26:35.553279       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:26:37 Starting Node Health Status refresh
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:26:38 Cleaned 0 nodes from health cache
>Sep 02 03:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:27:01.387624   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:27:01.388597   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 07:27:11.413685352 +0000 UTC
>Sep 02 03:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:13.524Z] [INFO] Validating specs...
>Sep 02 03:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:13.524Z] [NOTICE] All specs passed validation!
>Sep 02 03:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:27:13.911Z] [INFO] Validating specs...
>Sep 02 03:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:13.911Z] [NOTICE] All specs passed validation!
>Sep 02 03:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:27:14.309Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:28:09.201693   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:28:37 Starting Node Health Status refresh
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:28:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:28:37 Cleaned 0 nodes from health cache
>Sep 02 03:28:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:28:55.777Z] [INFO] AnsibleBroker::Catalog
>Sep 02 03:28:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:07:28:55 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 03:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:29:07.054058       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:30:01.464349   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:30:01.465561   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:30:37 Starting Node Health Status refresh
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:30:38 Cleaned 0 nodes from health cache
>Sep 02 03:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:32:37 Starting Node Health Status refresh
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:32:38 Cleaned 0 nodes from health cache
>Sep 02 03:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:33:01.504796   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:33:01.506007   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:33:09.202907   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:34:37 Starting Node Health Status refresh
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:34:38 Cleaned 0 nodes from health cache
>Sep 02 03:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:35:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:35:16.389579       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:35:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:35:39.630502       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:36:01.570013   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:36:01.571379   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:36:37 Starting Node Health Status refresh
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:36:38 Cleaned 0 nodes from health cache
>Sep 02 03:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 07:37:11.413898134 +0000 UTC
>Sep 02 03:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:13.291Z] [INFO] Validating specs...
>Sep 02 03:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:13.291Z] [NOTICE] All specs passed validation!
>Sep 02 03:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:37:13.672Z] [INFO] Validating specs...
>Sep 02 03:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:13.672Z] [NOTICE] All specs passed validation!
>Sep 02 03:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:37:13.881Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Job `cron.daily' started
>Sep 02 03:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12333]: starting logrotate
>Sep 02 03:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com rsyslogd[15009]:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="15009" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
>Sep 02 03:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12349]: finished logrotate
>Sep 02 03:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12351]: starting man-db.cron
>Sep 02 03:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12360]: finished man-db.cron
>Sep 02 03:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12362]: starting rhsmd
>Sep 02 03:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.daily)[12372]: finished rhsmd
>Sep 02 03:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Job `cron.daily' terminated
>Sep 02 03:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com anacron[8705]: Normal exit (1 job run)
>Sep 02 03:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 03:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 03:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:38:09.209472   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:38:37 Starting Node Health Status refresh
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:38:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:38:37 Cleaned 0 nodes from health cache
>Sep 02 03:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:39:01.590048   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:39:01.591135   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:40:37 Starting Node Health Status refresh
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:40:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:40:37 Cleaned 0 nodes from health cache
>Sep 02 03:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:42:01.630519   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:42:01.631879   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:42:37 Starting Node Health Status refresh
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:42:38 Cleaned 0 nodes from health cache
>Sep 02 03:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:43:09.213505   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:43:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:43:55.775Z] [INFO] AnsibleBroker::Catalog
>Sep 02 03:43:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:07:43:55 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 03:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:44:37 Starting Node Health Status refresh
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:44:38 Cleaned 0 nodes from health cache
>Sep 02 03:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:45:01.667834   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:45:01.668829   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:45:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:45:58.393746       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:46:37 Starting Node Health Status refresh
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:46:38 Cleaned 0 nodes from health cache
>Sep 02 03:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 07:47:11.414312551 +0000 UTC
>Sep 02 03:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:13.443Z] [INFO] Validating specs...
>Sep 02 03:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:13.444Z] [NOTICE] All specs passed validation!
>Sep 02 03:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:47:13.833Z] [INFO] Validating specs...
>Sep 02 03:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:13.833Z] [NOTICE] All specs passed validation!
>Sep 02 03:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:47:14.126Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:48:01.748830   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:48:01.750762   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:48:02.426202       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:48:09.214719   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:48:37 Starting Node Health Status refresh
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:48:38 Cleaned 0 nodes from health cache
>Sep 02 03:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:49:05.660734       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:50:37 Starting Node Health Status refresh
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:50:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:50:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:50:38 Cleaned 0 nodes from health cache
>Sep 02 03:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:51:01.785625   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:51:01.786679   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:52:37 Starting Node Health Status refresh
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:52:38 Cleaned 0 nodes from health cache
>Sep 02 03:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:53:09.216033   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:54:01.841275   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:54:01.842282   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:54:04.424209       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:54:37 Starting Node Health Status refresh
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:54:38 Cleaned 0 nodes from health cache
>Sep 02 03:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:56:37 Starting Node Health Status refresh
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:56:38 Cleaned 0 nodes from health cache
>Sep 02 03:56:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 07:56:59.534080       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 03:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:57:01.910049   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 03:57:01.911968   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 03:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 03:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 07:57:11.413652374 +0000 UTC
>Sep 02 03:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 03:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:13.612Z] [INFO] Validating specs...
>Sep 02 03:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:13.612Z] [NOTICE] All specs passed validation!
>Sep 02 03:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T07:57:14.003Z] [INFO] Validating specs...
>Sep 02 03:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:14.004Z] [NOTICE] All specs passed validation!
>Sep 02 03:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:57:14.445Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 03:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 03:58:09.217316   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 03:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:58:37 Starting Node Health Status refresh
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 03:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 07:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 07:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 03:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 07:58:38 Cleaned 0 nodes from health cache
>Sep 02 03:58:52 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 64.113.44.54 replaced with 75.102.39.146
>Sep 02 03:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 03:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T07:59:03.877Z] [INFO] AnsibleBroker::Catalog
>Sep 02 03:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:07:59:03 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 03:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 03:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 03:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 03:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 03:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:00:01.941992   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:00:01.942977   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 04:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 04:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:00:37 Starting Node Health Status refresh
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:00:38 Cleaned 0 nodes from health cache
>Sep 02 04:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 50 of user root.
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 50 of user root.
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[14680]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[14683]: starting 0anacron
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[14689]: finished 0anacron
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 04:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 04:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:02:37 Starting Node Health Status refresh
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:02:38 Cleaned 0 nodes from health cache
>Sep 02 04:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:03:01.979756   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:03:01.982753   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:03:09.220628   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:03:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:03:50.690539       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:04:37 Starting Node Health Status refresh
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:04:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:04:37 Cleaned 0 nodes from health cache
>Sep 02 04:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:06:02.045188   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:06:02.046325   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:06:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:06:23.567186       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:06:37 Starting Node Health Status refresh
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:06:38 Cleaned 0 nodes from health cache
>Sep 02 04:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 08:07:11.413714196 +0000 UTC
>Sep 02 04:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:13.359Z] [INFO] Validating specs...
>Sep 02 04:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:13.359Z] [NOTICE] All specs passed validation!
>Sep 02 04:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:07:13.824Z] [INFO] Validating specs...
>Sep 02 04:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:13.824Z] [NOTICE] All specs passed validation!
>Sep 02 04:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:07:14.364Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:08:09.223267   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:08:37 Starting Node Health Status refresh
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:08:38 Cleaned 0 nodes from health cache
>Sep 02 04:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:09:02.110570   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:09:02.111436   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:10:37 Starting Node Health Status refresh
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 22h ago
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 22h ago
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 22h ago
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:10:38 Cleaned 0 nodes from health cache
>Sep 02 04:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:11:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:11:13.527443       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:12:02.185778   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:12:02.187271   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:12:37 Starting Node Health Status refresh
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:12:38 Cleaned 0 nodes from health cache
>Sep 02 04:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:13:09.224646   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:14:03.87Z] [INFO] AnsibleBroker::Catalog
>Sep 02 04:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:08:14:03 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 04:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:14:37 Starting Node Health Status refresh
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:14:38 Cleaned 0 nodes from health cache
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:15:00.657122       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:15:02.216457   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:15:02.217896   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:16:37 Starting Node Health Status refresh
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:16:38 Cleaned 0 nodes from health cache
>Sep 02 04:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 08:17:11.414609222 +0000 UTC
>Sep 02 04:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:13.711Z] [INFO] Validating specs...
>Sep 02 04:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:13.711Z] [NOTICE] All specs passed validation!
>Sep 02 04:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:17:14.092Z] [INFO] Validating specs...
>Sep 02 04:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:14.092Z] [NOTICE] All specs passed validation!
>Sep 02 04:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:17:14.448Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:18:02.271032   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:18:02.271964   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:18:09.225706   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:18:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:18:29.797968       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:18:37 Starting Node Health Status refresh
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:18:38 Cleaned 0 nodes from health cache
>Sep 02 04:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:20:37 Starting Node Health Status refresh
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:20:38 Cleaned 0 nodes from health cache
>Sep 02 04:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:21:02.305567   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:21:02.306843   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:22:37 Starting Node Health Status refresh
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:22:38 Cleaned 0 nodes from health cache
>Sep 02 04:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:23:09.227008   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:24:02.364786   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:24:02.366308   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:24:37 Starting Node Health Status refresh
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:24:38 Cleaned 0 nodes from health cache
>Sep 02 04:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:24:39.691922       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:25:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:25:57.631120       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:26:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:26:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:26:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:26:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:26:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:26:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:26:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:26:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:26:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:26:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:26:37 Starting Node Health Status refresh
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:26:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:26:37 Cleaned 0 nodes from health cache
>Sep 02 04:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:27:02.387686   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:27:02.388604   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 08:27:11.41361298 +0000 UTC
>Sep 02 04:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:13.289Z] [INFO] Validating specs...
>Sep 02 04:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:13.29Z] [NOTICE] All specs passed validation!
>Sep 02 04:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:27:13.721Z] [INFO] Validating specs...
>Sep 02 04:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:13.722Z] [NOTICE] All specs passed validation!
>Sep 02 04:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:27:13.981Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:28:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:28:02.876307       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:28:09.228776   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:28:37 Starting Node Health Status refresh
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:28:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:28:37 Cleaned 0 nodes from health cache
>Sep 02 04:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:29:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:29:03.866Z] [INFO] AnsibleBroker::Catalog
>Sep 02 04:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:08:29:03 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 04:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:29:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:30:02.390689   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:30:02.391698   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:30:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:30:37 Starting Node Health Status refresh
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:30:38 Cleaned 0 nodes from health cache
>Sep 02 04:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:31:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:32:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:32:37 Starting Node Health Status refresh
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:32:38 Cleaned 0 nodes from health cache
>Sep 02 04:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:33:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:33:02.413688   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:33:02.414819   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:33:09.229977   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:33:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:34:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:34:37 Starting Node Health Status refresh
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:34:38 Cleaned 0 nodes from health cache
>Sep 02 04:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:35:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:35:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:35:10.704373       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:35:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:36:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:36:02.500420   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:36:02.501505   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:36:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:36:31.901940       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:36:37 Starting Node Health Status refresh
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:36:38 Cleaned 0 nodes from health cache
>Sep 02 04:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:37:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 08:37:11.414283286 +0000 UTC
>Sep 02 04:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:15.085Z] [INFO] Validating specs...
>Sep 02 04:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:15.085Z] [NOTICE] All specs passed validation!
>Sep 02 04:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:37:15.491Z] [INFO] Validating specs...
>Sep 02 04:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:15.492Z] [NOTICE] All specs passed validation!
>Sep 02 04:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:37:15.789Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:37:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:38:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:38:09.231262   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:38:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:38:37 Starting Node Health Status refresh
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:38:38 Cleaned 0 nodes from health cache
>Sep 02 04:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:39:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:39:02.557698   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:39:02.558723   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:39:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:40:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:40:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:40:10.773011       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:40:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:40:37 Starting Node Health Status refresh
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:40:38 Cleaned 0 nodes from health cache
>Sep 02 04:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:41:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:41:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:41:44 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 50.116.52.97 replaced with 154.16.245.246
>Sep 02 04:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:42:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:42:02.596447   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:42:02.597607   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:42:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:42:37 Starting Node Health Status refresh
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:42:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:42:37 Cleaned 0 nodes from health cache
>Sep 02 04:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:43:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:43:09.232450   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:43:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:44:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:44:03.865Z] [INFO] AnsibleBroker::Catalog
>Sep 02 04:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:08:44:03 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 04:44:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:44:20.742574       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:44:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:44:37 Starting Node Health Status refresh
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:44:38 Cleaned 0 nodes from health cache
>Sep 02 04:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:45:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:45:02.636506   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:45:02.637558   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:45:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:46:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:46:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:46:37 Starting Node Health Status refresh
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:46:38 Cleaned 0 nodes from health cache
>Sep 02 04:46:44 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Cleanup of Temporary Directories...
>Sep 02 04:46:44 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Cleanup of Temporary Directories.
>Sep 02 04:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:47:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 08:47:11.414547794 +0000 UTC
>Sep 02 04:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:13.463Z] [INFO] Validating specs...
>Sep 02 04:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:13.463Z] [NOTICE] All specs passed validation!
>Sep 02 04:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:47:13.876Z] [INFO] Validating specs...
>Sep 02 04:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:13.882Z] [NOTICE] All specs passed validation!
>Sep 02 04:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:47:14.463Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:47:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:48:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:48:02.685405   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:48:02.686320   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:48:09.233633   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:48:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:48:37 Starting Node Health Status refresh
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:48:38 Cleaned 0 nodes from health cache
>Sep 02 04:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:49:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:49:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:50:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:50:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:50:37 Starting Node Health Status refresh
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:50:38 Cleaned 0 nodes from health cache
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:51:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:51:00.942609       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:51:02.689859   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:51:02.691112   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:51:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:52:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:52:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:52:14.876749       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:52:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:52:37 Starting Node Health Status refresh
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:52:38 Cleaned 0 nodes from health cache
>Sep 02 04:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:53:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:53:09.234728   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:53:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:53:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 08:53:44.763030       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 04:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:54:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:54:02.701762   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:54:02.702847   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:54:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:54:37 Starting Node Health Status refresh
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:54:38 Cleaned 0 nodes from health cache
>Sep 02 04:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:55:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:55:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:56:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:56:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:56:37 Starting Node Health Status refresh
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:56:38 Cleaned 0 nodes from health cache
>Sep 02 04:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:57:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:57:02.778687   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 04:57:02.779629   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 04:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 04:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 08:57:11.413743271 +0000 UTC
>Sep 02 04:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 04:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:13.318Z] [INFO] Validating specs...
>Sep 02 04:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:13.319Z] [NOTICE] All specs passed validation!
>Sep 02 04:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T08:57:13.704Z] [INFO] Validating specs...
>Sep 02 04:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:13.705Z] [NOTICE] All specs passed validation!
>Sep 02 04:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:57:13.967Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 04:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:57:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:58:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 04:58:09.237343   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 04:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:58:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:58:37 Starting Node Health Status refresh
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 04:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 08:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 08:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 04:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 08:58:38 Cleaned 0 nodes from health cache
>Sep 02 04:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:59:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 04:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T08:59:03.871Z] [INFO] AnsibleBroker::Catalog
>Sep 02 04:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:08:59:03 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 04:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 04:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 04:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 04:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 04:59:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:00:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:00:02.866575   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:00:02.867495   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 05:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 05:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:00:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:00:37 Starting Node Health Status refresh
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:00:38 Cleaned 0 nodes from health cache
>Sep 02 05:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:00:38.966391       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:01:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 51 of user root.
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 51 of user root.
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[20775]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[20778]: starting 0anacron
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[20795]: finished 0anacron
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 05:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 05:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:01:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:02:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:02:37 Starting Node Health Status refresh
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:02:38 Cleaned 0 nodes from health cache
>Sep 02 05:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:03:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:03:02.895882   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:03:02.897588   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:03:09.240474   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:03:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:03:34.931738       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:04:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:04:37 Starting Node Health Status refresh
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:04:38 Cleaned 0 nodes from health cache
>Sep 02 05:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:05:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:05:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:06:02.917610   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:06:02.918752   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:06:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:06:37 Starting Node Health Status refresh
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:06:38 Cleaned 0 nodes from health cache
>Sep 02 05:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:07:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 09:07:11.413511779 +0000 UTC
>Sep 02 05:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:13.38Z] [INFO] Validating specs...
>Sep 02 05:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:13.38Z] [NOTICE] All specs passed validation!
>Sep 02 05:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:07:13.772Z] [INFO] Validating specs...
>Sep 02 05:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:13.772Z] [NOTICE] All specs passed validation!
>Sep 02 05:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:07:14.199Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:08:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:08:01.844787       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:08:09.242083   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:08:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:08:37 Starting Node Health Status refresh
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:08:38 Cleaned 0 nodes from health cache
>Sep 02 05:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:09:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:09:03.003384   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:09:03.004463   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:09:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:09:34.020883       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:10:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:10:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:10:37 Starting Node Health Status refresh
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 1 day 23h ago
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 1 day 23h ago
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 1 day 23h ago
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:10:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:10:37 Cleaned 0 nodes from health cache
>Sep 02 05:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:11:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:11:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:12:03.009831   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:12:03.010916   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:12:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:12:37 Starting Node Health Status refresh
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:12:38 Cleaned 0 nodes from health cache
>Sep 02 05:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:13:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:13:09.243499   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:13:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:13:31.991301       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:14:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:14:37 Starting Node Health Status refresh
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:14:38 Cleaned 0 nodes from health cache
>Sep 02 05:14:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:14:43.073Z] [INFO] AnsibleBroker::Catalog
>Sep 02 05:14:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:09:14:43 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 05:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:15:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:15:03.078469   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:15:03.079584   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:15:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:15:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:15:41.937066       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:16:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:16:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:16:37 Starting Node Health Status refresh
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:16:38 Cleaned 0 nodes from health cache
>Sep 02 05:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:17:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 09:17:11.413539504 +0000 UTC
>Sep 02 05:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:13.156Z] [INFO] Validating specs...
>Sep 02 05:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:13.157Z] [NOTICE] All specs passed validation!
>Sep 02 05:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:17:13.558Z] [INFO] Validating specs...
>Sep 02 05:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:13.558Z] [NOTICE] All specs passed validation!
>Sep 02 05:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:17:13.892Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:17:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:18:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:18:03.169988   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:18:03.170954   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:18:09.244799   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:18:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:18:37 Starting Node Health Status refresh
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:18:38 Cleaned 0 nodes from health cache
>Sep 02 05:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:19:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:19:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:20:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:20:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:20:37 Starting Node Health Status refresh
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:20:38 Cleaned 0 nodes from health cache
>Sep 02 05:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:21:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:21:03.260676   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:21:03.262363   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:22:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:22:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 199.102.46.77 replaced with 198.50.238.156
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:22:37 Starting Node Health Status refresh
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:22:38 Cleaned 0 nodes from health cache
>Sep 02 05:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:22:57.137762       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:23:09.245896   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:23:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:23:16.093777       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:24:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:24:03.353189   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:24:03.354269   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:24:37 Starting Node Health Status refresh
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:24:38 Cleaned 0 nodes from health cache
>Sep 02 05:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:25:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:26:37 Starting Node Health Status refresh
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:26:38 Cleaned 0 nodes from health cache
>Sep 02 05:26:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:26:42.992603       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:27:03.395120   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:27:03.396093   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 09:27:11.413607515 +0000 UTC
>Sep 02 05:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:14.209Z] [INFO] Validating specs...
>Sep 02 05:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:14.21Z] [NOTICE] All specs passed validation!
>Sep 02 05:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:27:14.589Z] [INFO] Validating specs...
>Sep 02 05:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:14.59Z] [NOTICE] All specs passed validation!
>Sep 02 05:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:27:15.048Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:28:09.248458   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:28:37 Starting Node Health Status refresh
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:28:38 Cleaned 0 nodes from health cache
>Sep 02 05:28:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:28:51.196740       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:30:03.460467   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:30:03.461517   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:30:37 Starting Node Health Status refresh
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:30:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:30:37 Cleaned 0 nodes from health cache
>Sep 02 05:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:32:37 Starting Node Health Status refresh
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com rsyslogd[15009]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:32:38 Cleaned 0 nodes from health cache
>Sep 02 05:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:33:03.476866   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:33:03.477848   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:33:08.204275       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:33:09.249626   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:33:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:33:45.134Z] [INFO] AnsibleBroker::Catalog
>Sep 02 05:33:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:09:33:45 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 05:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:34:37 Starting Node Health Status refresh
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:34:38 Cleaned 0 nodes from health cache
>Sep 02 05:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:35:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:35:12.105085       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:36:03.518650   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:36:03.519808   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:36:37 Starting Node Health Status refresh
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:36:38 Cleaned 0 nodes from health cache
>Sep 02 05:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 09:37:11.414260657 +0000 UTC
>Sep 02 05:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:13.319Z] [INFO] Validating specs...
>Sep 02 05:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:13.32Z] [NOTICE] All specs passed validation!
>Sep 02 05:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:37:13.713Z] [INFO] Validating specs...
>Sep 02 05:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:13.713Z] [NOTICE] All specs passed validation!
>Sep 02 05:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:37:13.913Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:38:09.250970   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:38:37 Starting Node Health Status refresh
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:38:38 Cleaned 0 nodes from health cache
>Sep 02 05:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:39:03.613010   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:39:03.613909   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:39:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:39:39.305197       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:40:37 Starting Node Health Status refresh
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:40:38 Cleaned 0 nodes from health cache
>Sep 02 05:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:42:03.713529   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:42:03.714507   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:42:37 Starting Node Health Status refresh
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:42:38 Cleaned 0 nodes from health cache
>Sep 02 05:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:43:09.252436   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:43:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:43:42.217345       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:44:37 Starting Node Health Status refresh
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:44:38 Cleaned 0 nodes from health cache
>Sep 02 05:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:45:03.814818   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:45:03.817580   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:46:37 Starting Node Health Status refresh
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:46:38 Cleaned 0 nodes from health cache
>Sep 02 05:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 09:47:11.413533211 +0000 UTC
>Sep 02 05:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:13.651Z] [INFO] Validating specs...
>Sep 02 05:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:13.652Z] [NOTICE] All specs passed validation!
>Sep 02 05:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:47:14.047Z] [INFO] Validating specs...
>Sep 02 05:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:14.048Z] [NOTICE] All specs passed validation!
>Sep 02 05:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:47:14.231Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:48:03.910641   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:48:03.911976   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:48:09.257742   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:48:37 Starting Node Health Status refresh
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:48:38 Cleaned 0 nodes from health cache
>Sep 02 05:48:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:48:45.125Z] [INFO] AnsibleBroker::Catalog
>Sep 02 05:48:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:09:48:45 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 05:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:49:53 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:49:53.180673       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:50:37 Starting Node Health Status refresh
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:50:38 Cleaned 0 nodes from health cache
>Sep 02 05:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:51:03.943737   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:51:03.944876   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:52:37 Starting Node Health Status refresh
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:52:38 Cleaned 0 nodes from health cache
>Sep 02 05:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:53:09.260385   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:53:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:53:28.411405       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:54:03.970336   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:54:03.971110   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:54:37 Starting Node Health Status refresh
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:54:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:54:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:54:38 Cleaned 0 nodes from health cache
>Sep 02 05:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:56:37 Starting Node Health Status refresh
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:56:38 Cleaned 0 nodes from health cache
>Sep 02 05:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:57:04.041001   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 05:57:04.041885   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 05:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 05:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 09:57:11.413602423 +0000 UTC
>Sep 02 05:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 05:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:13.411Z] [INFO] Validating specs...
>Sep 02 05:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:13.411Z] [NOTICE] All specs passed validation!
>Sep 02 05:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T09:57:13.81Z] [INFO] Validating specs...
>Sep 02 05:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:13.81Z] [NOTICE] All specs passed validation!
>Sep 02 05:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T09:57:14.445Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 05:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 05:58:09.262304   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 05:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:58:37 Starting Node Health Status refresh
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 05:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 09:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 09:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 05:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 09:58:38 Cleaned 0 nodes from health cache
>Sep 02 05:58:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 09:58:45.277222       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 05:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 05:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 05:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 05:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 05:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 05:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:00:04.064293   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:00:04.065201   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 06:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 06:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:00:37 Starting Node Health Status refresh
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:00:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:00:37 Cleaned 0 nodes from health cache
>Sep 02 06:00:44 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 45.63.111.145 replaced with 209.208.79.69
>Sep 02 06:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:00:59.439972       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 52 of user root.
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 52 of user root.
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[26767]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[26770]: starting 0anacron
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[26776]: finished 0anacron
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 06:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 06:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:02:37 Starting Node Health Status refresh
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:02:38 Cleaned 0 nodes from health cache
>Sep 02 06:02:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:02:44.254944       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:03:04.083461   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:03:04.085141   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:03:09.263828   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:03:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:03:45.131Z] [INFO] AnsibleBroker::Catalog
>Sep 02 06:03:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:10:03:45 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 06:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:04:37 Starting Node Health Status refresh
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:04:38 Cleaned 0 nodes from health cache
>Sep 02 06:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:06:04.130002   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:06:04.131166   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:06:37 Starting Node Health Status refresh
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:06:38 Cleaned 0 nodes from health cache
>Sep 02 06:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 10:07:11.413637577 +0000 UTC
>Sep 02 06:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:13.126Z] [INFO] Validating specs...
>Sep 02 06:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:13.126Z] [NOTICE] All specs passed validation!
>Sep 02 06:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:07:13.554Z] [INFO] Validating specs...
>Sep 02 06:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:13.554Z] [NOTICE] All specs passed validation!
>Sep 02 06:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:07:13.92Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:08:09.264994   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:08:37 Starting Node Health Status refresh
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:08:38 Cleaned 0 nodes from health cache
>Sep 02 06:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:09:04.179707   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:09:04.180707   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:09:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:09:12.490191       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:09:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:09:24.308964       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:10:37 Starting Node Health Status refresh
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:10:38 Cleaned 0 nodes from health cache
>Sep 02 06:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:12:04.220309   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:12:04.221252   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:12:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:12:17.328269       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:12:37 Starting Node Health Status refresh
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:12:38 Cleaned 0 nodes from health cache
>Sep 02 06:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:13:09.266845   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:14:37 Starting Node Health Status refresh
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:14:38 Cleaned 0 nodes from health cache
>Sep 02 06:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:15:04.267452   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:15:04.268910   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:16:37 Starting Node Health Status refresh
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:16:38 Cleaned 0 nodes from health cache
>Sep 02 06:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 10:17:11.413579397 +0000 UTC
>Sep 02 06:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:13.17Z] [INFO] Validating specs...
>Sep 02 06:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:13.17Z] [NOTICE] All specs passed validation!
>Sep 02 06:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:17:13.574Z] [INFO] Validating specs...
>Sep 02 06:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:13.574Z] [NOTICE] All specs passed validation!
>Sep 02 06:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:17:13.741Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:18:04.357182   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:18:04.358046   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:18:09.268105   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:18:37 Starting Node Health Status refresh
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:18:38 Cleaned 0 nodes from health cache
>Sep 02 06:18:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:18:43.559930       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:20:37 Starting Node Health Status refresh
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:20:38 Cleaned 0 nodes from health cache
>Sep 02 06:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:21:04.423154   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:21:04.424268   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:22:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:22:23.553205       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:22:37 Starting Node Health Status refresh
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:22:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:22:37 Cleaned 0 nodes from health cache
>Sep 02 06:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:23:09.269699   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:23:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:23:11.226Z] [INFO] AnsibleBroker::Catalog
>Sep 02 06:23:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:10:23:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 06:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:23:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:23:52.389803       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:24:04.441531   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:24:04.442694   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:24:37 Starting Node Health Status refresh
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:24:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:24:37 Cleaned 0 nodes from health cache
>Sep 02 06:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:26:37 Starting Node Health Status refresh
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:26:38 Cleaned 0 nodes from health cache
>Sep 02 06:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:27:04.449196   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:27:04.450112   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 10:27:11.413630072 +0000 UTC
>Sep 02 06:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:13.311Z] [INFO] Validating specs...
>Sep 02 06:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:13.311Z] [NOTICE] All specs passed validation!
>Sep 02 06:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:27:13.762Z] [INFO] Validating specs...
>Sep 02 06:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:13.763Z] [NOTICE] All specs passed validation!
>Sep 02 06:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:27:13.984Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:28:09.270918   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:28:37 Starting Node Health Status refresh
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:28:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:28:37 Cleaned 0 nodes from health cache
>Sep 02 06:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:30:04.496827   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:30:04.497661   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:30:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:30:15.620102       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:30:37 Starting Node Health Status refresh
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:30:38 Cleaned 0 nodes from health cache
>Sep 02 06:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:32:37 Starting Node Health Status refresh
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:32:38 Cleaned 0 nodes from health cache
>Sep 02 06:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com rhnsd[908]: /etc/sysconfig/rhn/systemid does not exist or is unreadable
>Sep 02 06:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:33:04.595549   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:33:04.596767   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:33:09.272607   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:33:31.675143       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:34:13 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 75.102.39.146 replaced with 204.11.201.12
>Sep 02 06:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:34:37 Starting Node Health Status refresh
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:34:38 Cleaned 0 nodes from health cache
>Sep 02 06:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:35:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:35:48.531114       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:36:04.601789   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:36:04.603642   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:36:37 Starting Node Health Status refresh
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:36:38 Cleaned 0 nodes from health cache
>Sep 02 06:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 10:37:11.413603073 +0000 UTC
>Sep 02 06:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:13.405Z] [INFO] Validating specs...
>Sep 02 06:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:13.407Z] [NOTICE] All specs passed validation!
>Sep 02 06:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:37:13.785Z] [INFO] Validating specs...
>Sep 02 06:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:13.785Z] [NOTICE] All specs passed validation!
>Sep 02 06:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:37:14.015Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:37:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:38:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:38:09.275755   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:38:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:38:37 Starting Node Health Status refresh
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:38:38 Cleaned 0 nodes from health cache
>Sep 02 06:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:39:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:39:04.615216   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:39:04.616527   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:39:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:40:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:40:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:40:13.949093       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:40:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:40:25.27Z] [INFO] AnsibleBroker::Catalog
>Sep 02 06:40:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:10:40:25 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 06:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:40:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:40:37 Starting Node Health Status refresh
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:40:38 Cleaned 0 nodes from health cache
>Sep 02 06:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:41:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:41:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:42:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:42:04.628988   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:42:04.629909   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:42:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:42:37 Starting Node Health Status refresh
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:42:38 Cleaned 0 nodes from health cache
>Sep 02 06:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:43:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:43:09.277985   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:43:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:44:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:44:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:44:13.718206       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:44:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:44:37 Starting Node Health Status refresh
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:44:38 Cleaned 0 nodes from health cache
>Sep 02 06:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:45:04.665965   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:45:04.667190   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:45:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:46:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:46:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:46:37 Starting Node Health Status refresh
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:46:38 Cleaned 0 nodes from health cache
>Sep 02 06:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:47:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 10:47:11.413652222 +0000 UTC
>Sep 02 06:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:13.468Z] [INFO] Validating specs...
>Sep 02 06:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:13.468Z] [NOTICE] All specs passed validation!
>Sep 02 06:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:47:13.858Z] [INFO] Validating specs...
>Sep 02 06:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:13.859Z] [NOTICE] All specs passed validation!
>Sep 02 06:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:47:14.013Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:47:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:48:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:48:04.745010   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:48:04.745910   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:48:09.279485   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:48:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:48:37 Starting Node Health Status refresh
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:48:38 Cleaned 0 nodes from health cache
>Sep 02 06:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:49:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:49:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:49:11.594400       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:49:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:50:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:50:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:50:37 Starting Node Health Status refresh
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:50:38 Cleaned 0 nodes from health cache
>Sep 02 06:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:51:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:51:04.808068   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:51:04.809005   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:51:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:52:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:52:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:52:37 Starting Node Health Status refresh
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:52:38 Cleaned 0 nodes from health cache
>Sep 02 06:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:53:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:53:09.280583   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:53:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:53:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:53:56.862177       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:54:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:54:04.905338   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:54:04.906079   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:54:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 10:54:34.019884       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:54:37 Starting Node Health Status refresh
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:54:38 Cleaned 0 nodes from health cache
>Sep 02 06:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:55:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:55:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:56:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:56:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:56:37 Starting Node Health Status refresh
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:56:38 Cleaned 0 nodes from health cache
>Sep 02 06:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:57:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:57:04.969641   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 06:57:04.970648   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 06:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 06:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 10:57:11.413623657 +0000 UTC
>Sep 02 06:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 06:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:13.62Z] [INFO] Validating specs...
>Sep 02 06:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:13.622Z] [NOTICE] All specs passed validation!
>Sep 02 06:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T10:57:14.057Z] [INFO] Validating specs...
>Sep 02 06:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:14.057Z] [NOTICE] All specs passed validation!
>Sep 02 06:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T10:57:14.27Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 06:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:57:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:58:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 06:58:09.281822   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 06:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:58:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:58:37 Starting Node Health Status refresh
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 06:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 10:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 10:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 06:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 10:58:38 Cleaned 0 nodes from health cache
>Sep 02 06:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:59:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 06:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 06:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 06:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 06:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 06:59:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:00:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:00:04.988414   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:00:04.989952   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:00:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:00:12.437Z] [INFO] AnsibleBroker::Catalog
>Sep 02 07:00:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:11:00:12 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 07:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 07:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 07:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:00:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:00:37 Starting Node Health Status refresh
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:00:38 Cleaned 0 nodes from health cache
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 53 of user root.
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 53 of user root.
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[32652]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[32655]: starting 0anacron
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[32661]: finished 0anacron
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 07:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 07:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:01:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:02:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:02:37 Starting Node Health Status refresh
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:02:38 Cleaned 0 nodes from health cache
>Sep 02 07:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:03:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:03:05.007419   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:03:05.008680   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:03:09.283908   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:03:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:04:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:04:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:04:13.699770       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:04:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:04:18.106032       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:04:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:04:37 Starting Node Health Status refresh
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:04:38 Cleaned 0 nodes from health cache
>Sep 02 07:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:05:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:05:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:06:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:06:05.077577   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:06:05.078646   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:06:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:06:37 Starting Node Health Status refresh
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:06:38 Cleaned 0 nodes from health cache
>Sep 02 07:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:07:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:07:11.413534477 +0000 UTC
>Sep 02 07:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:13.404Z] [INFO] Validating specs...
>Sep 02 07:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:13.405Z] [NOTICE] All specs passed validation!
>Sep 02 07:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:07:13.787Z] [INFO] Validating specs...
>Sep 02 07:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:13.788Z] [NOTICE] All specs passed validation!
>Sep 02 07:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:07:14.147Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:07:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:08:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:08:09.285811   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:08:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:08:37 Starting Node Health Status refresh
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:08:38 Cleaned 0 nodes from health cache
>Sep 02 07:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:09:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:09:05.112289   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:09:05.113143   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:09:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:09:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:09:58.907244       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:10:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:10:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:10:37 Starting Node Health Status refresh
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:10:38 Cleaned 0 nodes from health cache
>Sep 02 07:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:11:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:11:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:12:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:12:05.178554   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:12:05.179936   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:12:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:12:37 Starting Node Health Status refresh
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:12:38 Cleaned 0 nodes from health cache
>Sep 02 07:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:13:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:13:09.287143   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:13:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:14:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:14:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:14:15.157870       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:14:37 Starting Node Health Status refresh
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:14:38 Cleaned 0 nodes from health cache
>Sep 02 07:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:15:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:15:05.245753   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:15:05.246834   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:15:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:15:14.6Z] [INFO] AnsibleBroker::Catalog
>Sep 02 07:15:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:11:15:14 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 07:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:15:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:16:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:16:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:16:37 Starting Node Health Status refresh
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:16:38 Cleaned 0 nodes from health cache
>Sep 02 07:16:41 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 154.16.245.246 replaced with 166.70.136.35
>Sep 02 07:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:17:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:17:11.413563459 +0000 UTC
>Sep 02 07:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:13.301Z] [INFO] Validating specs...
>Sep 02 07:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:13.301Z] [NOTICE] All specs passed validation!
>Sep 02 07:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:17:13.827Z] [INFO] Validating specs...
>Sep 02 07:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:13.827Z] [NOTICE] All specs passed validation!
>Sep 02 07:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:17:14.018Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:17:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:18:05.284889   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:18:05.285723   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:18:09.288495   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:18:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:18:37 Starting Node Health Status refresh
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:18:38 Cleaned 0 nodes from health cache
>Sep 02 07:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:19:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:19:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:19:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:19:43.747018       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:20:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:20:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:20:37 Starting Node Health Status refresh
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:20:38 Cleaned 0 nodes from health cache
>Sep 02 07:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:21:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:21:05.385072   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:21:05.386076   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:21:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:22:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:22:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:22:12.034757       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:22:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:22:37 Starting Node Health Status refresh
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:22:38 Cleaned 0 nodes from health cache
>Sep 02 07:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:23:09.289697   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:24:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:24:05.391720   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:24:05.391907   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:24:37 Starting Node Health Status refresh
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:24:38 Cleaned 0 nodes from health cache
>Sep 02 07:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:26:37 Starting Node Health Status refresh
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:26:38 Cleaned 0 nodes from health cache
>Sep 02 07:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:27:05.410736   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:27:05.411851   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:27:11.413606011 +0000 UTC
>Sep 02 07:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:13.431Z] [INFO] Validating specs...
>Sep 02 07:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:13.431Z] [NOTICE] All specs passed validation!
>Sep 02 07:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:27:13.828Z] [INFO] Validating specs...
>Sep 02 07:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:13.828Z] [NOTICE] All specs passed validation!
>Sep 02 07:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:27:14.395Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:28:09.291120   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:28:37 Starting Node Health Status refresh
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:28:38 Cleaned 0 nodes from health cache
>Sep 02 07:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:29:59.228625       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:30:05.439180   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:30:05.440361   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:30:37 Starting Node Health Status refresh
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:30:38 Cleaned 0 nodes from health cache
>Sep 02 07:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:31:35.866967       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:31:36.066717       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:32:37 Starting Node Health Status refresh
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:32:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:32:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:32:38 Cleaned 0 nodes from health cache
>Sep 02 07:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:33:05.460692   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:33:05.461648   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:33:09.293011   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:33:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:33:42.67Z] [INFO] AnsibleBroker::Catalog
>Sep 02 07:33:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:11:33:42 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 07:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:34:37 Starting Node Health Status refresh
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:34:38 Cleaned 0 nodes from health cache
>Sep 02 07:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:36:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:36:05.525638   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:36:05.526542   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:36:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:36:37 Starting Node Health Status refresh
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:36:38 Cleaned 0 nodes from health cache
>Sep 02 07:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:37:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:37:11.413594186 +0000 UTC
>Sep 02 07:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:13.171Z] [INFO] Validating specs...
>Sep 02 07:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:13.173Z] [NOTICE] All specs passed validation!
>Sep 02 07:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:37:13.576Z] [INFO] Validating specs...
>Sep 02 07:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:13.576Z] [NOTICE] All specs passed validation!
>Sep 02 07:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:37:13.824Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:38:09.294283   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:38:37 Starting Node Health Status refresh
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:38:38 Cleaned 0 nodes from health cache
>Sep 02 07:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:39:05.560756   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:39:05.562053   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:40:37 Starting Node Health Status refresh
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:40:38 Cleaned 0 nodes from health cache
>Sep 02 07:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:42:05.631374   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:42:05.632540   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:42:37 Starting Node Health Status refresh
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:42:38 Cleaned 0 nodes from health cache
>Sep 02 07:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:43:06.307835       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:43:09.295584   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:43:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:43:54.178824       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:44:37 Starting Node Health Status refresh
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:44:38 Cleaned 0 nodes from health cache
>Sep 02 07:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:45:05.710153   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:45:05.711168   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:45:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:45:10.020382       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:46:37 Starting Node Health Status refresh
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:46:38 Cleaned 0 nodes from health cache
>Sep 02 07:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:47:11.413601741 +0000 UTC
>Sep 02 07:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:13.377Z] [INFO] Validating specs...
>Sep 02 07:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:13.377Z] [NOTICE] All specs passed validation!
>Sep 02 07:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:47:13.771Z] [INFO] Validating specs...
>Sep 02 07:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:13.771Z] [NOTICE] All specs passed validation!
>Sep 02 07:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:47:13.99Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:48:05.753982   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:48:05.755204   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:48:09.296813   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:48:37 Starting Node Health Status refresh
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:48:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:48:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:48:38 Cleaned 0 nodes from health cache
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:49:02.74Z] [INFO] AnsibleBroker::Catalog
>Sep 02 07:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:11:49:02 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 07:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:50:37 Starting Node Health Status refresh
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:50:38 Cleaned 0 nodes from health cache
>Sep 02 07:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:51:05.810150   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:51:05.811172   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:52:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:52:26.334340       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:52:37 Starting Node Health Status refresh
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:52:38 Cleaned 0 nodes from health cache
>Sep 02 07:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:53:09.298072   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:54:05.829600   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:54:05.830507   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:54:37 Starting Node Health Status refresh
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:54:38 Cleaned 0 nodes from health cache
>Sep 02 07:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:56:37 Starting Node Health Status refresh
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:56:38 Cleaned 0 nodes from health cache
>Sep 02 07:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:57:05.888388   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 07:57:05.889337   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 07:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 07:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 11:57:11.413545704 +0000 UTC
>Sep 02 07:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 07:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:13.407Z] [INFO] Validating specs...
>Sep 02 07:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:13.407Z] [NOTICE] All specs passed validation!
>Sep 02 07:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T11:57:13.811Z] [INFO] Validating specs...
>Sep 02 07:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:13.811Z] [NOTICE] All specs passed validation!
>Sep 02 07:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T11:57:13.981Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 07:57:29 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 198.50.238.156 replaced with 129.250.35.250
>Sep 02 07:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:57:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:57:47.111958       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 07:58:09.299517   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 07:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:58:37 Starting Node Health Status refresh
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 07:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 11:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 11:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 07:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 11:58:38 Cleaned 0 nodes from health cache
>Sep 02 07:58:53 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 11:58:53.225544       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 07:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 07:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 07:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 07:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 07:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 07:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:00:05.956804   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:00:05.957660   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 08:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 08:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:00:37 Starting Node Health Status refresh
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:00:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:00:37 Cleaned 0 nodes from health cache
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 54 of user root.
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 54 of user root.
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[6107]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[6110]: starting 0anacron
>Sep 02 08:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[6116]: finished 0anacron
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:02:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:02:26.357596       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:02:37 Starting Node Health Status refresh
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:02:38 Cleaned 0 nodes from health cache
>Sep 02 08:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:03:06.038001   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:03:06.039571   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:03:09.300769   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:04:02.737Z] [INFO] AnsibleBroker::Catalog
>Sep 02 08:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:12:04:02 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 08:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:04:37 Starting Node Health Status refresh
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:04:38 Cleaned 0 nodes from health cache
>Sep 02 08:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:06:06.120165   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:06:06.121381   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:06:37 Starting Node Health Status refresh
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:06:38 Cleaned 0 nodes from health cache
>Sep 02 08:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 12:07:11.413564213 +0000 UTC
>Sep 02 08:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:13.633Z] [INFO] Validating specs...
>Sep 02 08:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:13.633Z] [NOTICE] All specs passed validation!
>Sep 02 08:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:07:14.023Z] [INFO] Validating specs...
>Sep 02 08:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:14.023Z] [NOTICE] All specs passed validation!
>Sep 02 08:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:07:14.396Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:07:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:07:23.170484       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:08:09.302127   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:08:37 Starting Node Health Status refresh
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:08:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:08:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:08:38 Cleaned 0 nodes from health cache
>Sep 02 08:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:09:06.182907   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:09:06.183753   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:10:37 Starting Node Health Status refresh
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:10:38 Cleaned 0 nodes from health cache
>Sep 02 08:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:11:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:11:48.333340       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:12:06.246215   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:12:06.247438   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:12:37 Starting Node Health Status refresh
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:12:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:12:37 Cleaned 0 nodes from health cache
>Sep 02 08:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:13:09.303411   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:14:37 Starting Node Health Status refresh
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:14:38 Cleaned 0 nodes from health cache
>Sep 02 08:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:15:06.323794   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:15:06.325283   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:16:35.195363       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:16:37 Starting Node Health Status refresh
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:16:38 Cleaned 0 nodes from health cache
>Sep 02 08:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 12:17:11.413602722 +0000 UTC
>Sep 02 08:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com auditd[408]: Audit daemon rotating log files
>Sep 02 08:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:13.541Z] [INFO] Validating specs...
>Sep 02 08:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:13.541Z] [NOTICE] All specs passed validation!
>Sep 02 08:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:17:13.934Z] [INFO] Validating specs...
>Sep 02 08:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:13.934Z] [NOTICE] All specs passed validation!
>Sep 02 08:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:17:14.111Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:17:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:17:38.516360       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:18:06.374818   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:18:06.376487   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:18:09.307787   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:18:37 Starting Node Health Status refresh
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:18:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:18:37 Cleaned 0 nodes from health cache
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:19:02.791Z] [INFO] AnsibleBroker::Catalog
>Sep 02 08:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:12:19:02 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 08:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:20:36.429647       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:20:37 Starting Node Health Status refresh
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:20:38 Cleaned 0 nodes from health cache
>Sep 02 08:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:21:06.417736   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:21:06.418721   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:22:37 Starting Node Health Status refresh
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:22:38 Cleaned 0 nodes from health cache
>Sep 02 08:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:23:09.309816   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:24:06.467124   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:24:06.468061   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:24:37 Starting Node Health Status refresh
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:24:38 Cleaned 0 nodes from health cache
>Sep 02 08:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:26:37 Starting Node Health Status refresh
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:26:38 Cleaned 0 nodes from health cache
>Sep 02 08:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:27:06.472765   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:27:06.474005   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 12:27:11.413717571 +0000 UTC
>Sep 02 08:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:13.257Z] [INFO] Validating specs...
>Sep 02 08:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:13.259Z] [NOTICE] All specs passed validation!
>Sep 02 08:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:27:13.669Z] [INFO] Validating specs...
>Sep 02 08:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:13.67Z] [NOTICE] All specs passed validation!
>Sep 02 08:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:27:14.106Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:28:09.311142   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:28:33.285150       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:28:37 Starting Node Health Status refresh
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:28:38 Cleaned 0 nodes from health cache
>Sep 02 08:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:28:48.509996       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:29:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:29:25.578683       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:30:06.546500   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:30:06.547524   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:30:37 Starting Node Health Status refresh
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:30:38 Cleaned 0 nodes from health cache
>Sep 02 08:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:32:37 Starting Node Health Status refresh
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:32:38 Cleaned 0 nodes from health cache
>Sep 02 08:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:33:06.645175   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:33:06.646183   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:33:09.312442   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:34:02.739Z] [INFO] AnsibleBroker::Catalog
>Sep 02 08:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:12:34:02 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 08:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:34:37 Starting Node Health Status refresh
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:34:38 Cleaned 0 nodes from health cache
>Sep 02 08:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 209.208.79.69 replaced with 69.164.198.192
>Sep 02 08:35:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:35:49.369594       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:36:06.708521   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:36:06.709797   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:36:37 Starting Node Health Status refresh
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:36:38 Cleaned 0 nodes from health cache
>Sep 02 08:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 12:37:11.413513245 +0000 UTC
>Sep 02 08:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:13.537Z] [INFO] Validating specs...
>Sep 02 08:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:13.537Z] [NOTICE] All specs passed validation!
>Sep 02 08:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:37:13.975Z] [INFO] Validating specs...
>Sep 02 08:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:13.977Z] [NOTICE] All specs passed validation!
>Sep 02 08:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:37:14.31Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:38:09.313580   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:38:37 Starting Node Health Status refresh
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:38:38 Cleaned 0 nodes from health cache
>Sep 02 08:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:39:06.805906   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:39:06.806758   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:40:37 Starting Node Health Status refresh
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:40:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:40:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:40:38 Cleaned 0 nodes from health cache
>Sep 02 08:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:42:06.863544   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:42:06.864837   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:42:37 Starting Node Health Status refresh
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:42:38 Cleaned 0 nodes from health cache
>Sep 02 08:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:43:09.314948   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:44:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:44:08.402236       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:44:37 Starting Node Health Status refresh
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:44:38 Cleaned 0 nodes from health cache
>Sep 02 08:45:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:45:01.615186       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:45:06.870417   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:45:06.871376   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:46:37 Starting Node Health Status refresh
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:46:38 Cleaned 0 nodes from health cache
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:47:02.638216       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 12:47:11.413576283 +0000 UTC
>Sep 02 08:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:13.457Z] [INFO] Validating specs...
>Sep 02 08:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:13.459Z] [NOTICE] All specs passed validation!
>Sep 02 08:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:47:13.846Z] [INFO] Validating specs...
>Sep 02 08:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:13.846Z] [NOTICE] All specs passed validation!
>Sep 02 08:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:47:14.487Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:48:06.907978   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:48:06.908890   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:48:09.316475   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:48:37 Starting Node Health Status refresh
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:48:38 Cleaned 0 nodes from health cache
>Sep 02 08:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:49:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:49:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:50:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:50:05.875Z] [INFO] AnsibleBroker::Catalog
>Sep 02 08:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:12:50:05 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 08:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:50:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:50:37 Starting Node Health Status refresh
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:50:38 Cleaned 0 nodes from health cache
>Sep 02 08:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:51:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:51:06.916260   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:51:06.917393   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:51:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:52:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:52:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:52:37 Starting Node Health Status refresh
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:52:38 Cleaned 0 nodes from health cache
>Sep 02 08:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:53:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:53:09.317975   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:53:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:53:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:53:51.515648       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:54:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:54:06.935940   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:54:06.937008   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:54:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:54:37 Starting Node Health Status refresh
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:54:38 Cleaned 0 nodes from health cache
>Sep 02 08:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:55:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:55:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:56:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:56:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:56:37 Starting Node Health Status refresh
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:56:38 Cleaned 0 nodes from health cache
>Sep 02 08:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:57:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:57:06.997540   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 08:57:06.998599   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 08:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 08:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 12:57:11.413583657 +0000 UTC
>Sep 02 08:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 08:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:13.522Z] [INFO] Validating specs...
>Sep 02 08:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:13.524Z] [NOTICE] All specs passed validation!
>Sep 02 08:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T12:57:13.915Z] [INFO] Validating specs...
>Sep 02 08:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:13.915Z] [NOTICE] All specs passed validation!
>Sep 02 08:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T12:57:14.327Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 08:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:57:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:58:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 08:58:09.319139   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 08:58:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 12:58:15.640560       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 08:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:58:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:58:37 Starting Node Health Status refresh
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 08:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 12:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 12:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 08:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 12:58:38 Cleaned 0 nodes from health cache
>Sep 02 08:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:59:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 08:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 08:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 08:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 08:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 08:59:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:00:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:00:05.741100       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:00:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:00:07.084165   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:00:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:00:07.085054   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 09:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 09:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:00:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:00:37 Starting Node Health Status refresh
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:00:38 Cleaned 0 nodes from health cache
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 55 of user root.
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 55 of user root.
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[12004]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[12007]: starting 0anacron
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[12013]: finished 0anacron
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 09:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 09:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:01:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:01:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:02:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:02:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:02:37 Starting Node Health Status refresh
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:02:38 Cleaned 0 nodes from health cache
>Sep 02 09:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:03:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:03:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:03:07.230831   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:03:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:03:07.232715   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:03:09.320966   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:03:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:04:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:04:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:04:37 Starting Node Health Status refresh
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:04:38 Cleaned 0 nodes from health cache
>Sep 02 09:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:05:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:05:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:05:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:05:55.605208       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:06:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:06:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:06:07.304769   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:06:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:06:07.306394   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:06:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:06:37 Starting Node Health Status refresh
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:06:38 Cleaned 0 nodes from health cache
>Sep 02 09:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:07:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:07:11.413579806 +0000 UTC
>Sep 02 09:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:13.215Z] [INFO] Validating specs...
>Sep 02 09:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:13.216Z] [NOTICE] All specs passed validation!
>Sep 02 09:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:07:13.6Z] [INFO] Validating specs...
>Sep 02 09:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:13.6Z] [NOTICE] All specs passed validation!
>Sep 02 09:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:07:13.726Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:07:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:08:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:08:09.322985   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:08:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:08:12.667832       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:08:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:08:37 Starting Node Health Status refresh
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:08:38 Cleaned 0 nodes from health cache
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 204.11.201.12 replaced with 74.6.168.72
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:09:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:09:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:09:07.309442   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:09:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:09:07.310460   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:09:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:09:17.055Z] [INFO] AnsibleBroker::Catalog
>Sep 02 09:09:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:13:09:17 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 09:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:10:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:10:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:10:37 Starting Node Health Status refresh
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:10:38 Cleaned 0 nodes from health cache
>Sep 02 09:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:11:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:11:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:12:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:12:04.828568       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:12:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:12:07.384084   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:12:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:12:07.385401   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:12:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:12:37 Starting Node Health Status refresh
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:12:38 Cleaned 0 nodes from health cache
>Sep 02 09:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:13:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:13:09.324184   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:13:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:14:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:14:37 Starting Node Health Status refresh
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:14:38 Cleaned 0 nodes from health cache
>Sep 02 09:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:15:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:15:04.688217       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:15:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:15:07.424793   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:15:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:15:07.425645   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:15:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:16:37 Starting Node Health Status refresh
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:16:38 Cleaned 0 nodes from health cache
>Sep 02 09:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:17:11.413765634 +0000 UTC
>Sep 02 09:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:13.661Z] [INFO] Validating specs...
>Sep 02 09:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:13.661Z] [NOTICE] All specs passed validation!
>Sep 02 09:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:17:14.907Z] [INFO] Validating specs...
>Sep 02 09:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:14.907Z] [NOTICE] All specs passed validation!
>Sep 02 09:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:17:15.138Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:17:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:17:17.726393       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:18:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:18:07.496890   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:18:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:18:07.497731   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:18:09.325478   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:18:37 Starting Node Health Status refresh
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:18:38 Cleaned 0 nodes from health cache
>Sep 02 09:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:20:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:20:37 Starting Node Health Status refresh
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:20:38 Cleaned 0 nodes from health cache
>Sep 02 09:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:21:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:21:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:21:07.504622   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:21:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:21:07.505800   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:22:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:22:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:22:37 Starting Node Health Status refresh
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:22:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:22:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:22:39 Cleaned 0 nodes from health cache
>Sep 02 09:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:23:09.326686   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:23:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:23:55.168395       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:24:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:24:07.516896   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:24:07.517952   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:24:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:24:22.830893       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:24:32.747771       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:24:37 Starting Node Health Status refresh
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:24:38 Cleaned 0 nodes from health cache
>Sep 02 09:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:25:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:25:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:26:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:26:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:26:12.101Z] [INFO] AnsibleBroker::Catalog
>Sep 02 09:26:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:13:26:12 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 09:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:26:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:26:37 Starting Node Health Status refresh
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:26:38 Cleaned 0 nodes from health cache
>Sep 02 09:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:27:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:27:07.588093   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:27:07.589172   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:27:11.413529647 +0000 UTC
>Sep 02 09:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:13.404Z] [INFO] Validating specs...
>Sep 02 09:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:13.404Z] [NOTICE] All specs passed validation!
>Sep 02 09:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:27:13.804Z] [INFO] Validating specs...
>Sep 02 09:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:13.804Z] [NOTICE] All specs passed validation!
>Sep 02 09:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:27:14.379Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:27:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:28:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:28:09.328022   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:28:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:28:37 Starting Node Health Status refresh
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:28:38 Cleaned 0 nodes from health cache
>Sep 02 09:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:29:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:29:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:30:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:30:07.612197   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:30:07.613583   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:30:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:30:37 Starting Node Health Status refresh
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:30:38 Cleaned 0 nodes from health cache
>Sep 02 09:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:31:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:32:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:32:37 Starting Node Health Status refresh
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:32:38 Cleaned 0 nodes from health cache
>Sep 02 09:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:33:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:33:07.616183   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:33:07.617126   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:33:09.329291   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:34:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:34:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:34:37 Starting Node Health Status refresh
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:34:38 Cleaned 0 nodes from health cache
>Sep 02 09:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:35:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:35:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:36:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:36:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:36:07.701832   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:36:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:36:07.703033   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:36:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:36:37 Starting Node Health Status refresh
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:36:38 Cleaned 0 nodes from health cache
>Sep 02 09:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:37:11.413587677 +0000 UTC
>Sep 02 09:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:13.425Z] [INFO] Validating specs...
>Sep 02 09:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:13.426Z] [NOTICE] All specs passed validation!
>Sep 02 09:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:37:13.815Z] [INFO] Validating specs...
>Sep 02 09:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:13.815Z] [NOTICE] All specs passed validation!
>Sep 02 09:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:37:14.094Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:37:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:37:22.857152       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:37:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:38:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:38:09.330788   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:38:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:38:18.262267       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:38:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:38:37 Starting Node Health Status refresh
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:38:38 Cleaned 0 nodes from health cache
>Sep 02 09:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:39:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:39:07.773185   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:39:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:39:07.774152   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:39:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:40:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:40:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:40:37 Starting Node Health Status refresh
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:40:38 Cleaned 0 nodes from health cache
>Sep 02 09:40:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:40:43.907839       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:41:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:41:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:42:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:42:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:42:07.870544   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:42:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:42:07.871666   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:42:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:42:22.591Z] [INFO] AnsibleBroker::Catalog
>Sep 02 09:42:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:13:42:22 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 09:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:42:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:42:37 Starting Node Health Status refresh
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:42:38 Cleaned 0 nodes from health cache
>Sep 02 09:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:43:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:43:09.332042   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:43:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:44:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:44:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:44:37 Starting Node Health Status refresh
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:44:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:44:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:44:38 Cleaned 0 nodes from health cache
>Sep 02 09:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:45:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:45:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:45:07.975964   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:45:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:45:07.977005   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:45:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:46:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:46:05.334620       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:46:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:46:37 Starting Node Health Status refresh
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:46:38 Cleaned 0 nodes from health cache
>Sep 02 09:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:47:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:47:11.413625106 +0000 UTC
>Sep 02 09:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:14.357Z] [INFO] Validating specs...
>Sep 02 09:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:14.357Z] [NOTICE] All specs passed validation!
>Sep 02 09:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:47:14.751Z] [INFO] Validating specs...
>Sep 02 09:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:14.761Z] [NOTICE] All specs passed validation!
>Sep 02 09:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:47:15.313Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:47:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:47:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:47:58.650071       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:48:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:48:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:48:08.008781   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:48:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:48:08.009633   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:48:09.333261   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:48:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:48:37 Starting Node Health Status refresh
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:48:38 Cleaned 0 nodes from health cache
>Sep 02 09:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:50:37 Starting Node Health Status refresh
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:50:38 Cleaned 0 nodes from health cache
>Sep 02 09:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:51:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:51:08.099533   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:51:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:51:08.100682   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:52:07 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 166.70.136.35 replaced with 2607:f740:1::42c
>Sep 02 09:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:52:37 Starting Node Health Status refresh
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:52:38 Cleaned 0 nodes from health cache
>Sep 02 09:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:53:09.334541   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:53:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:53:47.696728       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:54:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:54:08.119502   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:54:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:54:08.120365   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:54:37 Starting Node Health Status refresh
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:54:38 Cleaned 0 nodes from health cache
>Sep 02 09:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:54:38.385888       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 13:55:36.991743       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 09:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:56:37 Starting Node Health Status refresh
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:56:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:56:39 Cleaned 0 nodes from health cache
>Sep 02 09:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:57:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:57:08.121311   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:57:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 09:57:08.122507   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 09:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 09:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 13:57:11.413573281 +0000 UTC
>Sep 02 09:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 09:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:13.414Z] [INFO] Validating specs...
>Sep 02 09:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:13.415Z] [NOTICE] All specs passed validation!
>Sep 02 09:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T13:57:13.813Z] [INFO] Validating specs...
>Sep 02 09:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:13.813Z] [NOTICE] All specs passed validation!
>Sep 02 09:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T13:57:14.106Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 09:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 09:58:09.336279   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 09:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:58:37 Starting Node Health Status refresh
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 09:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 13:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 13:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 09:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 13:58:38 Cleaned 0 nodes from health cache
>Sep 02 09:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 09:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 09:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 09:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 09:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 09:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:00:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:00:08.217822   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:00:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:00:08.218766   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 10:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 10:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:00:37 Starting Node Health Status refresh
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:00:38 Cleaned 0 nodes from health cache
>Sep 02 10:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:00:59.632Z] [INFO] AnsibleBroker::Catalog
>Sep 02 10:00:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:14:00:59 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 56 of user root.
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 56 of user root.
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[17973]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[17976]: starting 0anacron
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[17982]: finished 0anacron
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 10:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 10:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:02:37 Starting Node Health Status refresh
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:02:38 Cleaned 0 nodes from health cache
>Sep 02 10:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:03:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:03:08.318745   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:03:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:03:08.321028   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:03:09.339327   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:04:37 Starting Node Health Status refresh
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:04:38 Cleaned 0 nodes from health cache
>Sep 02 10:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:06:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:06:00.799435       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:06:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:06:08.373422   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:06:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:06:08.374343   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:06:37 Starting Node Health Status refresh
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:06:38 Cleaned 0 nodes from health cache
>Sep 02 10:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:11.415Z] [INFO] Attempting bootstrap at 2018-09-02 14:07:11.414362855 +0000 UTC
>Sep 02 10:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:13.718Z] [INFO] Validating specs...
>Sep 02 10:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:13.72Z] [NOTICE] All specs passed validation!
>Sep 02 10:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:07:14.107Z] [INFO] Validating specs...
>Sep 02 10:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:14.107Z] [NOTICE] All specs passed validation!
>Sep 02 10:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:07:14.748Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:07:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:07:49.456343       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:08:09.341181   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:08:37 Starting Node Health Status refresh
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:08:38 Cleaned 0 nodes from health cache
>Sep 02 10:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:09:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:09:08.429039   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:09:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:09:08.430061   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:10:06.035169       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:10:37 Starting Node Health Status refresh
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:10:38 Cleaned 0 nodes from health cache
>Sep 02 10:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:12:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:12:08.555956   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:12:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:12:08.557043   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:12:37 Starting Node Health Status refresh
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:12:38 Cleaned 0 nodes from health cache
>Sep 02 10:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:13:09.342349   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:14:37 Starting Node Health Status refresh
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:14:38 Cleaned 0 nodes from health cache
>Sep 02 10:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:15:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:15:08.568193       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:15:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:15:08.632867   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:15:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:15:08.634000   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:16:37 Starting Node Health Status refresh
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:16:38 Cleaned 0 nodes from health cache
>Sep 02 10:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 14:17:11.413632773 +0000 UTC
>Sep 02 10:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:13.534Z] [INFO] Validating specs...
>Sep 02 10:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:13.536Z] [NOTICE] All specs passed validation!
>Sep 02 10:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:17:13.959Z] [INFO] Validating specs...
>Sep 02 10:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:13.973Z] [NOTICE] All specs passed validation!
>Sep 02 10:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:14.217Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:17:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:17:39.738Z] [INFO] AnsibleBroker::Catalog
>Sep 02 10:17:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:14:17:39 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 10:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:18:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:18:08.730040   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:18:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:18:08.730967   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:18:09.343674   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:18:37 Starting Node Health Status refresh
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:18:38 Cleaned 0 nodes from health cache
>Sep 02 10:18:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:18:51.101937       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:19:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:19:25.871223       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:20:37 Starting Node Health Status refresh
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:20:38 Cleaned 0 nodes from health cache
>Sep 02 10:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:21:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:21:08.736516   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:21:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:21:08.738490   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:22:37 Starting Node Health Status refresh
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:22:38 Cleaned 0 nodes from health cache
>Sep 02 10:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:23:09.345048   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:24:08.832706   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:24:08.834145   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:24:37 Starting Node Health Status refresh
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:24:38 Cleaned 0 nodes from health cache
>Sep 02 10:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:26:37 Starting Node Health Status refresh
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:26:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:26:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:26:38 Cleaned 0 nodes from health cache
>Sep 02 10:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:27:08.928064   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:27:08.929205   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 14:27:11.413608999 +0000 UTC
>Sep 02 10:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:13.553Z] [INFO] Validating specs...
>Sep 02 10:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:13.553Z] [NOTICE] All specs passed validation!
>Sep 02 10:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:27:13.94Z] [INFO] Validating specs...
>Sep 02 10:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:13.941Z] [NOTICE] All specs passed validation!
>Sep 02 10:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:27:14.564Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:28:09.346281   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:28:37 Starting Node Health Status refresh
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:28:38 Cleaned 0 nodes from health cache
>Sep 02 10:28:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:28:57.640766       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:30:08.934274   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:30:08.935175   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:30:37 Starting Node Health Status refresh
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:30:38 Cleaned 0 nodes from health cache
>Sep 02 10:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:32:14 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 129.250.35.250 replaced with 138.68.46.177
>Sep 02 10:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:32:37 Starting Node Health Status refresh
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:32:38 Cleaned 0 nodes from health cache
>Sep 02 10:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com rhnsd[908]: /etc/sysconfig/rhn/systemid does not exist or is unreadable
>Sep 02 10:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:33:08.936134   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:33:08.937295   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:33:09.347932   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:33:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:33:47.210495       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:34:37 Starting Node Health Status refresh
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:34:38 Cleaned 0 nodes from health cache
>Sep 02 10:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:36:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:36:08.966024   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:36:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:36:08.966990   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:36:35.827Z] [INFO] AnsibleBroker::Catalog
>Sep 02 10:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:14:36:35 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:36:37 Starting Node Health Status refresh
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:36:38 Cleaned 0 nodes from health cache
>Sep 02 10:37:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:37:02.956126       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 14:37:11.413584152 +0000 UTC
>Sep 02 10:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:13.518Z] [INFO] Validating specs...
>Sep 02 10:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:13.518Z] [NOTICE] All specs passed validation!
>Sep 02 10:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:37:13.943Z] [INFO] Validating specs...
>Sep 02 10:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:13.943Z] [NOTICE] All specs passed validation!
>Sep 02 10:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:37:14.543Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:38:09.350172   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:38:37 Starting Node Health Status refresh
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:38:38 Cleaned 0 nodes from health cache
>Sep 02 10:38:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:38:49.689388       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:39:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:39:09.033219   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:39:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:39:09.034392   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:40:37 Starting Node Health Status refresh
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:40:38 Cleaned 0 nodes from health cache
>Sep 02 10:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:42:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:42:09.130984   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:42:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:42:09.132099   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:42:37 Starting Node Health Status refresh
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:42:38 Cleaned 0 nodes from health cache
>Sep 02 10:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:43:09.351319   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:44:37 Starting Node Health Status refresh
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:44:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:44:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:44:38 Cleaned 0 nodes from health cache
>Sep 02 10:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:45:05.023938       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:45:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:45:09.136481   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:45:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:45:09.137699   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:46:37 Starting Node Health Status refresh
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:46:38 Cleaned 0 nodes from health cache
>Sep 02 10:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 14:47:11.41353351 +0000 UTC
>Sep 02 10:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:13.643Z] [INFO] Validating specs...
>Sep 02 10:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:13.644Z] [NOTICE] All specs passed validation!
>Sep 02 10:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:47:14.08Z] [INFO] Validating specs...
>Sep 02 10:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:14.081Z] [NOTICE] All specs passed validation!
>Sep 02 10:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:47:14.621Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:47:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:47:52.269147       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:48:09.197400   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:48:09.198373   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:48:09.352611   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:48:37 Starting Node Health Status refresh
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:48:38 Cleaned 0 nodes from health cache
>Sep 02 10:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:50:37 Starting Node Health Status refresh
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:50:38 Cleaned 0 nodes from health cache
>Sep 02 10:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:51:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:51:09.238108   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:51:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:51:09.239369   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:52:37 Starting Node Health Status refresh
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:52:38 Cleaned 0 nodes from health cache
>Sep 02 10:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:53:09.354114   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:54:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:54:09.332837   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:54:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:54:09.333910   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:54:37 Starting Node Health Status refresh
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:54:38 Cleaned 0 nodes from health cache
>Sep 02 10:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:55:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:55:25.376043       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:56:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:56:22.776383       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:56:34.951Z] [INFO] AnsibleBroker::Catalog
>Sep 02 10:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:14:56:34 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 10:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:56:37 Starting Node Health Status refresh
>Sep 02 10:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:56:38 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:56:38 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:56:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:56:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:56:38 Cleaned 0 nodes from health cache
>Sep 02 10:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:57:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:57:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:57:09.434807   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:57:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 10:57:09.435927   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 10:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 10:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 14:57:11.413639113 +0000 UTC
>Sep 02 10:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 10:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:13.606Z] [INFO] Validating specs...
>Sep 02 10:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:13.606Z] [NOTICE] All specs passed validation!
>Sep 02 10:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T14:57:14.03Z] [INFO] Validating specs...
>Sep 02 10:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:14.03Z] [NOTICE] All specs passed validation!
>Sep 02 10:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T14:57:14.447Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 10:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:57:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:58:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 10:58:09.355431   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 10:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:58:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:58:37 Starting Node Health Status refresh
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 10:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 14:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 14:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 10:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 14:58:38 Cleaned 0 nodes from health cache
>Sep 02 10:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:59:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 14:59:33.110504       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 10:59:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:00:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:00:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:00:09.455843   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:00:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:00:09.456941   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 11:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 11:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:00:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:00:37 Starting Node Health Status refresh
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:00:38 Cleaned 0 nodes from health cache
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 57 of user root.
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 57 of user root.
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[24120]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[24123]: starting 0anacron
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[24129]: finished 0anacron
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 11:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 11:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:01:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:01:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:02:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:02:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:02:37 Starting Node Health Status refresh
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:02:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:02:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:02:39 Cleaned 0 nodes from health cache
>Sep 02 11:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:03:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:03:09.358032   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:03:09.461060   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:03:09.462107   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:03:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:04:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:04:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:04:37 Starting Node Health Status refresh
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:04:38 Cleaned 0 nodes from health cache
>Sep 02 11:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:05:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:05:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:05:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:05:56.855574       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:06:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:06:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:06:09.487734   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:06:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:06:09.488749   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:06:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:06:37 Starting Node Health Status refresh
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:06:38 Cleaned 0 nodes from health cache
>Sep 02 11:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:07:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:07:11.413599657 +0000 UTC
>Sep 02 11:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:13.553Z] [INFO] Validating specs...
>Sep 02 11:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:13.553Z] [NOTICE] All specs passed validation!
>Sep 02 11:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:07:13.958Z] [INFO] Validating specs...
>Sep 02 11:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:13.958Z] [NOTICE] All specs passed validation!
>Sep 02 11:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:07:14.233Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:07:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:08:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:08:09.360618   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:08:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:08:34.479757       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:08:37 Starting Node Health Status refresh
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:08:38 Cleaned 0 nodes from health cache
>Sep 02 11:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:09:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:09:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:09:09.541735   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:09:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:09:09.542605   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:09:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:10:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:10:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:10:37 Starting Node Health Status refresh
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:10:38 Cleaned 0 nodes from health cache
>Sep 02 11:10:57 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 69.164.198.192 replaced with 69.164.213.136
>Sep 02 11:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:11:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:11:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:12:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:12:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:12:09.631630   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:12:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:12:09.633595   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:12:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:12:37 Starting Node Health Status refresh
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:12:38 Cleaned 0 nodes from health cache
>Sep 02 11:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:13:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:13:09.362053   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:13:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:13:25.238069       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:13:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:14:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:14:37 Starting Node Health Status refresh
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:14:38 Cleaned 0 nodes from health cache
>Sep 02 11:14:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:14:43.057Z] [INFO] AnsibleBroker::Catalog
>Sep 02 11:14:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:15:14:43 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 11:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:15:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:15:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:15:09.686122   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:15:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:15:09.687062   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:15:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:16:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:16:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:16:21.502053       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:16:37 Starting Node Health Status refresh
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:16:38 Cleaned 0 nodes from health cache
>Sep 02 11:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:17:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:17:11.413554267 +0000 UTC
>Sep 02 11:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:13.476Z] [INFO] Validating specs...
>Sep 02 11:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:13.478Z] [NOTICE] All specs passed validation!
>Sep 02 11:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:17:13.864Z] [INFO] Validating specs...
>Sep 02 11:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:13.864Z] [NOTICE] All specs passed validation!
>Sep 02 11:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:17:14.314Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:17:36.946778       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:18:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:18:09.363121   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:18:09.747129   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:18:09.748881   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:18:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:18:37 Starting Node Health Status refresh
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:18:38 Cleaned 0 nodes from health cache
>Sep 02 11:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:19:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:19:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:20:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:20:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:20:37 Starting Node Health Status refresh
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:20:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:20:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:20:38 Cleaned 0 nodes from health cache
>Sep 02 11:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:21:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:21:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:21:09.834097   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:21:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:21:09.834991   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:22:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:22:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:22:37 Starting Node Health Status refresh
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:22:38 Cleaned 0 nodes from health cache
>Sep 02 11:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:22:47.316999       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:23:09.364438   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:23:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:24:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:24:09.931936   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:24:09.932981   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:24:37 Starting Node Health Status refresh
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:24:38 Cleaned 0 nodes from health cache
>Sep 02 11:24:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:24:52.556741       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:25:01.972463       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:26:37 Starting Node Health Status refresh
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:26:38 Cleaned 0 nodes from health cache
>Sep 02 11:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:27:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:27:09.949896   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:27:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:27:09.950958   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:27:11.413587829 +0000 UTC
>Sep 02 11:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:13.56Z] [INFO] Validating specs...
>Sep 02 11:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:13.56Z] [NOTICE] All specs passed validation!
>Sep 02 11:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:27:13.957Z] [INFO] Validating specs...
>Sep 02 11:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:13.957Z] [NOTICE] All specs passed validation!
>Sep 02 11:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:27:14.175Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:28:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:28:09.365776   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:28:37 Starting Node Health Status refresh
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:28:38 Cleaned 0 nodes from health cache
>Sep 02 11:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:30:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:30:10.045894   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:30:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:30:10.046970   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:30:37 Starting Node Health Status refresh
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:30:38 Cleaned 0 nodes from health cache
>Sep 02 11:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:32:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:32:10.403458       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:32:35.144Z] [INFO] AnsibleBroker::Catalog
>Sep 02 11:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:15:32:35 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:32:37 Starting Node Health Status refresh
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:32:38 Cleaned 0 nodes from health cache
>Sep 02 11:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:33:09.367116   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:33:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:33:10.135212   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:33:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:33:10.136137   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:34:37 Starting Node Health Status refresh
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:34:38 Cleaned 0 nodes from health cache
>Sep 02 11:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:36:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:36:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:36:10.173186   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:36:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:36:10.174489   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:36:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:36:37 Starting Node Health Status refresh
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:36:38 Cleaned 0 nodes from health cache
>Sep 02 11:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:37:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:37:11.413646537 +0000 UTC
>Sep 02 11:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:13.331Z] [INFO] Validating specs...
>Sep 02 11:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:13.331Z] [NOTICE] All specs passed validation!
>Sep 02 11:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:37:13.7Z] [INFO] Validating specs...
>Sep 02 11:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:13.7Z] [NOTICE] All specs passed validation!
>Sep 02 11:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:37:14.061Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:37:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:38:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:38:09.369115   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:38:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:38:37 Starting Node Health Status refresh
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:38:38 Cleaned 0 nodes from health cache
>Sep 02 11:38:53 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:38:53.054183       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:39:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:39:02.671368       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:39:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:39:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:39:10.268701   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:39:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:39:10.270691   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:39:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:40:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:40:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:40:37 Starting Node Health Status refresh
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:40:38 Cleaned 0 nodes from health cache
>Sep 02 11:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:41:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:41:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:41:15.424300       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:41:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:42:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:42:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:42:10.344613   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:42:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:42:10.345792   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:42:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:42:37 Starting Node Health Status refresh
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:42:38 Cleaned 0 nodes from health cache
>Sep 02 11:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:43:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:43:09.370314   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:43:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:44:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 74.6.168.72 replaced with 103.105.49.148
>Sep 02 11:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:44:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:44:37 Starting Node Health Status refresh
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:44:38 Cleaned 0 nodes from health cache
>Sep 02 11:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:45:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:45:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:45:10.438556   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:45:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:45:10.439616   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:45:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:46:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:46:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:46:37 Starting Node Health Status refresh
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:46:38 Cleaned 0 nodes from health cache
>Sep 02 11:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:47:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:47:11.413624255 +0000 UTC
>Sep 02 11:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:13.46Z] [INFO] Validating specs...
>Sep 02 11:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:13.462Z] [NOTICE] All specs passed validation!
>Sep 02 11:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:47:13.869Z] [INFO] Validating specs...
>Sep 02 11:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:13.869Z] [NOTICE] All specs passed validation!
>Sep 02 11:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:47:14.145Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:47:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:48:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:48:09.371576   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:48:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:48:10.535798   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:48:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:48:10.536609   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:48:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:48:37 Starting Node Health Status refresh
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:48:38 Cleaned 0 nodes from health cache
>Sep 02 11:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:49:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:49:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:49:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:49:42.211Z] [INFO] AnsibleBroker::Catalog
>Sep 02 11:49:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:15:49:42 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 11:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:50:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:50:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:50:37 Starting Node Health Status refresh
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:50:38 Cleaned 0 nodes from health cache
>Sep 02 11:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:51:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:51:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:51:10.539445   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:51:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:51:10.541034   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:51:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:52:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:52:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:52:37 Starting Node Health Status refresh
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:52:38 Cleaned 0 nodes from health cache
>Sep 02 11:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:53:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:53:09.372814   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:53:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:53:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:53:59.695003       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:54:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:54:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:54:10.565027   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:54:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:54:10.566081   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:54:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:54:37 Starting Node Health Status refresh
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:54:38 Cleaned 0 nodes from health cache
>Sep 02 11:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:55:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:55:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:55:26.446785       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:55:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 15:55:34.162559       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 11:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:56:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:56:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:56:37 Starting Node Health Status refresh
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:56:38 Cleaned 0 nodes from health cache
>Sep 02 11:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:57:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:57:10.641517   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:57:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 11:57:10.642942   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 11:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 11:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 15:57:11.413637829 +0000 UTC
>Sep 02 11:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 11:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:13.505Z] [INFO] Validating specs...
>Sep 02 11:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:13.507Z] [NOTICE] All specs passed validation!
>Sep 02 11:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T15:57:13.906Z] [INFO] Validating specs...
>Sep 02 11:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:13.907Z] [NOTICE] All specs passed validation!
>Sep 02 11:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T15:57:14.196Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 11:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 11:58:09.374355   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 11:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:58:37 Starting Node Health Status refresh
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 11:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 15:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 15:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 11:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 15:58:38 Cleaned 0 nodes from health cache
>Sep 02 11:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 11:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 11:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 11:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 11:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 11:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:00:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:00:10.742985   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:00:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:00:10.744177   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 12:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 12:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:00:37 Starting Node Health Status refresh
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:00:38 Cleaned 0 nodes from health cache
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 58 of user root.
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[30010]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 58 of user root.
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[30013]: starting 0anacron
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[30019]: finished 0anacron
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 12:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 12:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:02:37 Starting Node Health Status refresh
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:02:38 Cleaned 0 nodes from health cache
>Sep 02 12:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:03:09.377733   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:03:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:03:10.765964   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:03:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:03:10.766703   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:04:37 Starting Node Health Status refresh
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:04:38 Cleaned 0 nodes from health cache
>Sep 02 12:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:06:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:06:10.811704   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:06:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:06:10.812731   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:06:37 Starting Node Health Status refresh
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:06:38 Cleaned 0 nodes from health cache
>Sep 02 12:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:11.415Z] [INFO] Attempting bootstrap at 2018-09-02 16:07:11.413591353 +0000 UTC
>Sep 02 12:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:13.719Z] [INFO] Validating specs...
>Sep 02 12:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:13.719Z] [NOTICE] All specs passed validation!
>Sep 02 12:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:07:14.108Z] [INFO] Validating specs...
>Sep 02 12:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:14.108Z] [NOTICE] All specs passed validation!
>Sep 02 12:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:07:14.484Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:07:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:07:30.719841       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:08:09.380804   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:08:37 Starting Node Health Status refresh
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:08:38 Cleaned 0 nodes from health cache
>Sep 02 12:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:09:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:09:10.837071   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:09:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:09:10.838158   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:09:32.283Z] [INFO] AnsibleBroker::Catalog
>Sep 02 12:09:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:16:09:32 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 12:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:09:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:09:46.274635       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:10:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:10:29.552995       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:10:37 Starting Node Health Status refresh
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:10:38 Cleaned 0 nodes from health cache
>Sep 02 12:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:12:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:12:10.840408   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:12:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:12:10.841459   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:12:37 Starting Node Health Status refresh
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:12:38 Cleaned 0 nodes from health cache
>Sep 02 12:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:13:09.382043   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:14:37 Starting Node Health Status refresh
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:14:38 Cleaned 0 nodes from health cache
>Sep 02 12:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:15:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:15:10.875303   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:15:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:15:10.876397   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:16:37 Starting Node Health Status refresh
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:16:38 Cleaned 0 nodes from health cache
>Sep 02 12:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 16:17:11.41358937 +0000 UTC
>Sep 02 12:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:13.509Z] [INFO] Validating specs...
>Sep 02 12:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:13.51Z] [NOTICE] All specs passed validation!
>Sep 02 12:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:17:13.891Z] [INFO] Validating specs...
>Sep 02 12:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:13.891Z] [NOTICE] All specs passed validation!
>Sep 02 12:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:17:14.319Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:18:09.383376   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:18:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:18:10.936683   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:18:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:18:10.938276   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:18:13.749498       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:18:37 Starting Node Health Status refresh
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:18:38 Cleaned 0 nodes from health cache
>Sep 02 12:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:19:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:19:23.593978       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:20:37 Starting Node Health Status refresh
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:20:38 Cleaned 0 nodes from health cache
>Sep 02 12:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:21:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:21:10.976518   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:21:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:21:10.977563   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:22:37 Starting Node Health Status refresh
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:22:38 Cleaned 0 nodes from health cache
>Sep 02 12:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:23:09.384796   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:24:11.041541   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:24:11.042561   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:24:37 Starting Node Health Status refresh
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:24:38 Cleaned 0 nodes from health cache
>Sep 02 12:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:25:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:25:19.389464       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:25:59.405Z] [INFO] AnsibleBroker::Catalog
>Sep 02 12:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:16:25:59 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 12:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:26:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:26:18.806638       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:26:37 Starting Node Health Status refresh
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:26:38 Cleaned 0 nodes from health cache
>Sep 02 12:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 2607:f740:1::42c replaced with 173.71.68.101
>Sep 02 12:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:27:11.050188   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:27:11.051378   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 16:27:11.413633176 +0000 UTC
>Sep 02 12:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:13.443Z] [INFO] Validating specs...
>Sep 02 12:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:13.443Z] [NOTICE] All specs passed validation!
>Sep 02 12:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:27:13.84Z] [INFO] Validating specs...
>Sep 02 12:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:13.84Z] [NOTICE] All specs passed validation!
>Sep 02 12:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:27:15.115Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:28:09.386487   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:28:37 Starting Node Health Status refresh
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:28:38 Cleaned 0 nodes from health cache
>Sep 02 12:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:30:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:30:11.146682   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:30:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:30:11.147612   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:30:37 Starting Node Health Status refresh
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:30:38 Cleaned 0 nodes from health cache
>Sep 02 12:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:32:37 Starting Node Health Status refresh
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:32:38 Cleaned 0 nodes from health cache
>Sep 02 12:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:33:09.387719   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:33:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:33:11.237813   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:33:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:33:11.238661   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:34:15.713217       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:34:37 Starting Node Health Status refresh
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:34:38 Cleaned 0 nodes from health cache
>Sep 02 12:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:35:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:35:10.918757       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:36:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:36:11.285874   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:36:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:36:11.286967   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:36:37 Starting Node Health Status refresh
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:36:38 Cleaned 0 nodes from health cache
>Sep 02 12:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 16:37:11.413591668 +0000 UTC
>Sep 02 12:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:14.601Z] [INFO] Validating specs...
>Sep 02 12:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:14.601Z] [NOTICE] All specs passed validation!
>Sep 02 12:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:37:14.994Z] [INFO] Validating specs...
>Sep 02 12:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:14.996Z] [NOTICE] All specs passed validation!
>Sep 02 12:37:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:37:15.524Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:38:09.390141   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:38:37 Starting Node Health Status refresh
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:38:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:38:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:38:38 Cleaned 0 nodes from health cache
>Sep 02 12:38:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:38:52.423484       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:39:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:39:11.361491   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:39:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:39:11.362924   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:40:37 Starting Node Health Status refresh
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:40:38 Cleaned 0 nodes from health cache
>Sep 02 12:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:42:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:42:11.408365   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:42:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:42:11.409407   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:42:37 Starting Node Health Status refresh
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:42:38 Cleaned 0 nodes from health cache
>Sep 02 12:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:43:09.391455   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:44:28.487Z] [INFO] AnsibleBroker::Catalog
>Sep 02 12:44:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:16:44:28 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 12:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:44:37 Starting Node Health Status refresh
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:44:38 Cleaned 0 nodes from health cache
>Sep 02 12:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:45:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:45:11.468877   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:45:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:45:11.470043   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:46:37 Starting Node Health Status refresh
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:46:38 Cleaned 0 nodes from health cache
>Sep 02 12:46:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:46:59.745178       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 16:47:11.413784589 +0000 UTC
>Sep 02 12:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:47:13.975985       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:14.35Z] [INFO] Validating specs...
>Sep 02 12:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:14.35Z] [NOTICE] All specs passed validation!
>Sep 02 12:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:47:14.733Z] [INFO] Validating specs...
>Sep 02 12:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:14.734Z] [NOTICE] All specs passed validation!
>Sep 02 12:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:47:15.101Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:48:09.392881   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:48:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:48:11.552603   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:48:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:48:11.553432   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:48:37 Starting Node Health Status refresh
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:48:38 Cleaned 0 nodes from health cache
>Sep 02 12:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:50:37 Starting Node Health Status refresh
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:50:38 Cleaned 0 nodes from health cache
>Sep 02 12:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:51:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:51:11.650211   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:51:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:51:11.651275   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:52:37 Starting Node Health Status refresh
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:52:38 Cleaned 0 nodes from health cache
>Sep 02 12:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:53:09.394365   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:53:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:53:49.856136       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:54:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:54:11.664430   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:54:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:54:11.665579   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:54:37 Starting Node Health Status refresh
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:54:38 Cleaned 0 nodes from health cache
>Sep 02 12:54:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:54:40.043156       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 16:55:34.476685       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 12:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:56:37 Starting Node Health Status refresh
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:56:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:56:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:56:38 Cleaned 0 nodes from health cache
>Sep 02 12:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 12:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 16:57:11.413617208 +0000 UTC
>Sep 02 12:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 12:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:57:11.763180   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 12:57:11.764205   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 12:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:13.566Z] [INFO] Validating specs...
>Sep 02 12:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:13.567Z] [NOTICE] All specs passed validation!
>Sep 02 12:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T16:57:13.994Z] [INFO] Validating specs...
>Sep 02 12:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:13.994Z] [NOTICE] All specs passed validation!
>Sep 02 12:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T16:57:14.527Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 12:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 12:58:09.395536   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 12:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:58:37 Starting Node Health Status refresh
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 12:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 16:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 16:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 12:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 16:58:38 Cleaned 0 nodes from health cache
>Sep 02 12:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 12:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 12:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 12:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 12:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 12:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:00:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:00:11.845856   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:00:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:00:11.846920   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 13:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 13:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:00:37 Starting Node Health Status refresh
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:00:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:00:37 Cleaned 0 nodes from health cache
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 59 of user root.
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 59 of user root.
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[3468]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[3471]: starting 0anacron
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[3477]: finished 0anacron
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 13:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 13:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:02:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:02:11.648Z] [INFO] AnsibleBroker::Catalog
>Sep 02 13:02:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:17:02:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 13:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:02:37 Starting Node Health Status refresh
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:02:38 Cleaned 0 nodes from health cache
>Sep 02 13:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:03:09.397656   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:03:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:03:11.941248   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:03:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:03:11.943497   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:03:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:03:47.917683       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:04:37 Starting Node Health Status refresh
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:04:38 Cleaned 0 nodes from health cache
>Sep 02 13:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:06:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:06:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:06:12.035284   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:06:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:06:12.036708   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:06:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:06:37 Starting Node Health Status refresh
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:06:38 Cleaned 0 nodes from health cache
>Sep 02 13:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 17:07:11.413578868 +0000 UTC
>Sep 02 13:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:13.473Z] [INFO] Validating specs...
>Sep 02 13:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:13.473Z] [NOTICE] All specs passed validation!
>Sep 02 13:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:07:13.858Z] [INFO] Validating specs...
>Sep 02 13:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:13.858Z] [NOTICE] All specs passed validation!
>Sep 02 13:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:07:14.662Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:07:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:07:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:07:42.151200       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:07:42 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 138.68.46.177 replaced with 184.105.182.15
>Sep 02 13:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:08:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:08:09.400198   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:08:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:08:29.540555       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:08:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:08:37 Starting Node Health Status refresh
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:08:38 Cleaned 0 nodes from health cache
>Sep 02 13:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:09:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:09:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:09:12.065729   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:09:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:09:12.066660   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:09:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:10:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:10:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:10:37 Starting Node Health Status refresh
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:10:38 Cleaned 0 nodes from health cache
>Sep 02 13:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:11:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:11:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:12:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:12:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:12:12.151082   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:12:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:12:12.152288   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:12:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:12:37 Starting Node Health Status refresh
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:12:38 Cleaned 0 nodes from health cache
>Sep 02 13:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:13:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:13:09.402267   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:13:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:14:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:14:32.175558       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:14:37 Starting Node Health Status refresh
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:14:38 Cleaned 0 nodes from health cache
>Sep 02 13:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:15:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:15:12.239489   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:15:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:15:12.241496   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:15:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:16:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:16:37 Starting Node Health Status refresh
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:16:38 Cleaned 0 nodes from health cache
>Sep 02 13:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:17:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 17:17:11.414611256 +0000 UTC
>Sep 02 13:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:13.49Z] [INFO] Validating specs...
>Sep 02 13:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:13.49Z] [NOTICE] All specs passed validation!
>Sep 02 13:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:17:13.578752       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:17:13.88Z] [INFO] Validating specs...
>Sep 02 13:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:13.88Z] [NOTICE] All specs passed validation!
>Sep 02 13:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:17:14.119Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:17:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:17:42.024799       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:18:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:18:09.403491   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:18:12.273008   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:18:12.274016   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:18:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:18:37 Starting Node Health Status refresh
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:18:38 Cleaned 0 nodes from health cache
>Sep 02 13:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:19:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:19:28.708Z] [INFO] AnsibleBroker::Catalog
>Sep 02 13:19:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:17:19:28 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 13:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:20:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:20:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:20:37 Starting Node Health Status refresh
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:20:38 Cleaned 0 nodes from health cache
>Sep 02 13:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:21:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:21:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:21:12.369976   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:21:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:21:12.370995   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:22:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:22:37 Starting Node Health Status refresh
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:22:38 Cleaned 0 nodes from health cache
>Sep 02 13:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:23:09.404808   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:23:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:24:12.417962   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:24:12.418886   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:24:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:24:21.280317       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:24:37 Starting Node Health Status refresh
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:24:38 Cleaned 0 nodes from health cache
>Sep 02 13:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:25:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:25:31.097979       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:25:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:26:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:26:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:26:37 Starting Node Health Status refresh
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:26:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:26:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:26:38 Cleaned 0 nodes from health cache
>Sep 02 13:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 17:27:11.413598779 +0000 UTC
>Sep 02 13:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:27:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:27:12.464201   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:27:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:27:12.465151   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:13.206Z] [INFO] Validating specs...
>Sep 02 13:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:13.206Z] [NOTICE] All specs passed validation!
>Sep 02 13:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:27:13.609Z] [INFO] Validating specs...
>Sep 02 13:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:13.609Z] [NOTICE] All specs passed validation!
>Sep 02 13:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:27:14.015Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:27:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:28:09.406012   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:28:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:28:37 Starting Node Health Status refresh
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:28:38 Cleaned 0 nodes from health cache
>Sep 02 13:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:29:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:29:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:30:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:30:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:30:12.495070   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:30:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:30:12.496368   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:30:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:30:37 Starting Node Health Status refresh
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:30:38 Cleaned 0 nodes from health cache
>Sep 02 13:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:31:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:31:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:32:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:32:34.629708       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:32:37 Starting Node Health Status refresh
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:32:38 Cleaned 0 nodes from health cache
>Sep 02 13:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:33:09.407154   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:33:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:33:12.523631   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:33:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:33:12.524676   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:33:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:34:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:34:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:34:37 Starting Node Health Status refresh
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:34:38 Cleaned 0 nodes from health cache
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:34:38.747Z] [INFO] AnsibleBroker::Catalog
>Sep 02 13:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:17:34:38 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: DHCPREQUEST on eth0 to 10.10.94.229 port 67 (xid=0x5127a64f)
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: DHCPACK from 10.10.94.229 (xid=0x5127a64f)
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7849] dhcp4 (eth0):   address 10.10.94.27
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7868] dhcp4 (eth0):   plen 22 (255.255.252.0)
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7873] dhcp4 (eth0):   classless static route 169.254.169.254/32 gw 10.10.94.229
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7877] dhcp4 (eth0):   gateway 10.10.95.254
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7881] dhcp4 (eth0):   lease time 86400
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7889] dhcp4 (eth0):   hostname 'host-10-10-94-27'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7889] dhcp4 (eth0):   nameserver '10.11.5.4'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7889] dhcp4 (eth0):   nameserver '10.11.5.3'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7889] dhcp4 (eth0):   domain name 'openstacklocal'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535909686.7889] dhcp4 (eth0): state changed bound -> bound
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dbus[470]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Network Manager Script Dispatcher Service...
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dhclient[13533]: bound to 10.10.94.27 -- renewal in 35236 seconds.
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dbus[470]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Network Manager Script Dispatcher Service.
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: req:1 'dhcp4-change' [eth0]: new request (6 scripts)
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + cd /etc/sysconfig/network-scripts
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + . ./network-functions
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ export PATH
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ hostname
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ HOSTNAME=node-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ . /etc/init.d/functions
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ TEXTDOMAIN=initscripts
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ umask 022
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ export PATH
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' 6807 -ne 1 -a -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -d /run/systemd/system ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ case "$0" in
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ COLUMNS=80
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -c /dev/stderr -a -r /dev/stderr ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ CONSOLETYPE=serial
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ . /etc/profile.d/lang.sh
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ unset LANGSH_SOURCED
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -z '' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' -f /etc/sysconfig/init ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ . /etc/sysconfig/init
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ BOOTUP=color
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ RES_COL=60
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ MOVE_TO_COL='echo -en \033[60G'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ SETCOLOR_FAILURE='echo -en \033[0;31m'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ SETCOLOR_WARNING='echo -en \033[0;33m'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ SETCOLOR_NORMAL='echo -en \033[0;39m'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' serial = serial ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ BOOTUP=serial
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ MOVE_TO_COL=
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ SETCOLOR_SUCCESS=
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ SETCOLOR_FAILURE=
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ SETCOLOR_WARNING=
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ SETCOLOR_NORMAL=
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ __sed_discard_ignored_files='/$~\|\.bak\|\.old\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave$$/d'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' '' = 1 ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++++ cat /proc/cmdline
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ strstr 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' rc.debug
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ '[' 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' = 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=716e713d-4e91-4186-81fd-c6cfa1b0974d ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8' ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ return 1
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: +++ return 0
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + '[' -f ../network ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + . ../network
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ NETWORKING=yes
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ NOZEROCONF=yes
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + [[ dhcp4-change =~ ^(up|dhcp4-change|dhcp6-change)$ ]]
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + NEEDS_RESTART=0
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + UPSTREAM_DNS=/etc/dnsmasq.d/origin-upstream-dns.conf
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ mktemp
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + UPSTREAM_DNS_TMP=/tmp/tmp.ub8xtIqdKk
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ mktemp
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + UPSTREAM_DNS_TMP_SORTED=/tmp/tmp.Jy6Wr7EkJ4
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ mktemp
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + CURRENT_UPSTREAM_DNS_SORTED=/tmp/tmp.VuX7ddMgc8
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ mktemp
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + NEW_RESOLV_CONF=/tmp/tmp.0DxtGUG0sk
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ mktemp
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + NEW_NODE_RESOLV_CONF=/tmp/tmp.PNOStpTlTO
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ awk '{print $3 }'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ /sbin/ip route list match 0.0.0.0/0
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + def_route=10.10.95.254
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ awk '{print $3}'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ /sbin/ip route get to 10.10.95.254
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + def_route_int=eth0
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ awk '{print $5}'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ /sbin/ip route get to 10.10.95.254
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + def_route_ip=10.10.94.27
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + [[ eth0 == eth0 ]]
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + '[' '!' -f /etc/dnsmasq.d/origin-dns.conf ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + grep -q 99-origin-dns.sh /etc/resolv.conf
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ systemctl -q is-active dnsmasq.service
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + '[' 0 -eq 1 ']'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: ++ systemctl -q is-active dnsmasq.service
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + grep -q 99-origin-dns.sh /etc/resolv.conf
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + sed -e '/^nameserver.*$/d' /etc/resolv.conf
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + echo 'nameserver 10.10.94.27'
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + grep -qw search /tmp/tmp.0DxtGUG0sk
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + grep -q 'search cluster.local' /tmp/tmp.0DxtGUG0sk
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + cp -Z /tmp/tmp.0DxtGUG0sk /etc/resolv.conf
>Sep 02 13:34:46 node-0.alxcns2.lab.rdu2.cee.redhat.com nm-dispatcher[6807]: + rm -f /tmp/tmp.ub8xtIqdKk /tmp/tmp.Jy6Wr7EkJ4 /tmp/tmp.VuX7ddMgc8 /tmp/tmp.0DxtGUG0sk
>Sep 02 13:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:35:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:35:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:36:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:36:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:36:12.547741   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:36:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:36:12.549551   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:36:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:36:37 Starting Node Health Status refresh
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:36:38 Cleaned 0 nodes from health cache
>Sep 02 13:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:37:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 17:37:11.413606484 +0000 UTC
>Sep 02 13:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:13.297Z] [INFO] Validating specs...
>Sep 02 13:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:13.299Z] [NOTICE] All specs passed validation!
>Sep 02 13:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:37:13.741Z] [INFO] Validating specs...
>Sep 02 13:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:13.741Z] [NOTICE] All specs passed validation!
>Sep 02 13:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:37:14.53Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:37:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:38:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:38:06.176619       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:38:09.409435   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:38:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:38:37 Starting Node Health Status refresh
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:38:38 Cleaned 0 nodes from health cache
>Sep 02 13:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:39:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:39:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:39:12.642862   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:39:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:39:12.643893   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:39:34.326609       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:39:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:40:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:40:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:40:37 Starting Node Health Status refresh
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:40:38 Cleaned 0 nodes from health cache
>Sep 02 13:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:41:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:41:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:42:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:42:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:42:12.648081   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:42:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:42:12.649127   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:42:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:42:37 Starting Node Health Status refresh
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:42:38 Cleaned 0 nodes from health cache
>Sep 02 13:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:43:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:43:09.411251   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:43:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:44:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:44:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:44:37 Starting Node Health Status refresh
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:44:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:44:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:44:38 Cleaned 0 nodes from health cache
>Sep 02 13:44:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:44:56.248329       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:45:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:45:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:45:12.715375   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:45:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:45:12.716663   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:45:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:46:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 69.164.213.136 replaced with 72.5.72.15
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:46:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:46:37 Starting Node Health Status refresh
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:46:38 Cleaned 0 nodes from health cache
>Sep 02 13:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:47:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 17:47:11.414085707 +0000 UTC
>Sep 02 13:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:13.676Z] [INFO] Validating specs...
>Sep 02 13:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:13.676Z] [NOTICE] All specs passed validation!
>Sep 02 13:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:47:14.097Z] [INFO] Validating specs...
>Sep 02 13:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:14.098Z] [NOTICE] All specs passed validation!
>Sep 02 13:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:47:14.924Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:47:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:47:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:47:44.677551       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:48:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:48:09.412450   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:48:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:48:12.744041   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:48:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:48:12.745286   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:48:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:48:37 Starting Node Health Status refresh
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:48:38 Cleaned 0 nodes from health cache
>Sep 02 13:48:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:48:48.386509       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:49:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:49:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:49:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:49:38.743Z] [INFO] AnsibleBroker::Catalog
>Sep 02 13:49:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:17:49:38 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 13:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:50:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:50:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:50:37 Starting Node Health Status refresh
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:50:38 Cleaned 0 nodes from health cache
>Sep 02 13:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:51:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:51:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:51:12.758004   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:51:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:51:12.758894   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:51:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:52:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:52:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:52:37 Starting Node Health Status refresh
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:52:38 Cleaned 0 nodes from health cache
>Sep 02 13:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:53:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:53:09.413862   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:53:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:54:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:54:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:54:12.778676       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:54:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:54:12.810401   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:54:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:54:12.811393   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:54:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:54:37 Starting Node Health Status refresh
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:54:38 Cleaned 0 nodes from health cache
>Sep 02 13:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:55:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:55:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:56:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:56:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:56:37 Starting Node Health Status refresh
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:56:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:56:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:56:39 Cleaned 0 nodes from health cache
>Sep 02 13:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:57:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 13:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 17:57:11.41358218 +0000 UTC
>Sep 02 13:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 13:57:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:57:12.858866   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:57:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 13:57:12.859790   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 13:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:13.791Z] [INFO] Validating specs...
>Sep 02 13:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:13.791Z] [NOTICE] All specs passed validation!
>Sep 02 13:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T17:57:14.202Z] [INFO] Validating specs...
>Sep 02 13:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:14.202Z] [NOTICE] All specs passed validation!
>Sep 02 13:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T17:57:14.75Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 13:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:57:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:58:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 13:58:09.415086   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 13:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:58:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:58:37 Starting Node Health Status refresh
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 13:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 17:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 17:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 13:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 17:58:38 Cleaned 0 nodes from health cache
>Sep 02 13:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 13:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 17:59:05.371910       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 13:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 13:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 13:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 13:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 13:59:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:00:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:00:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:00:12.958152   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:00:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:00:12.959875   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 14:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 14:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:00:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:00:37 Starting Node Health Status refresh
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:00:38 Cleaned 0 nodes from health cache
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 60 of user root.
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[9412]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 60 of user root.
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[9415]: starting 0anacron
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[9421]: finished 0anacron
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 14:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 14:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:01:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:01:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:02:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:02:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:02:27.439723       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:02:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:02:37 Starting Node Health Status refresh
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:02:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:02:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:02:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:02:39 Cleaned 0 nodes from health cache
>Sep 02 14:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:03:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:03:09.417282   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:03:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:03:13.052123   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:03:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:03:13.053106   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:03:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:04:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:04:00.883944       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:04:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:04:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:04:37 Starting Node Health Status refresh
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:04:38 Cleaned 0 nodes from health cache
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:04:38.747Z] [INFO] AnsibleBroker::Catalog
>Sep 02 14:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:18:04:38 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 14:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:05:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:05:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:06:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:06:13.082366   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:06:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:06:13.083509   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:06:37 Starting Node Health Status refresh
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:06:38 Cleaned 0 nodes from health cache
>Sep 02 14:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 18:07:11.413572121 +0000 UTC
>Sep 02 14:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:13.472Z] [INFO] Validating specs...
>Sep 02 14:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:13.473Z] [NOTICE] All specs passed validation!
>Sep 02 14:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:07:13.833Z] [INFO] Validating specs...
>Sep 02 14:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:13.833Z] [NOTICE] All specs passed validation!
>Sep 02 14:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:07:14.23Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:08:09.421449   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:08:37 Starting Node Health Status refresh
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:08:38 Cleaned 0 nodes from health cache
>Sep 02 14:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:09:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:09:13.140105   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:09:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:09:13.141062   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:10:37 Starting Node Health Status refresh
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:10:38 Cleaned 0 nodes from health cache
>Sep 02 14:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:12:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:12:13.144620   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:12:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:12:13.146173   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:12:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:12:18.519180       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:12:37 Starting Node Health Status refresh
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:12:38 Cleaned 0 nodes from health cache
>Sep 02 14:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:13:09.422747   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:14:37 Starting Node Health Status refresh
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:14:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:14:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:14:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:14:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:14:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:14:39 Cleaned 0 nodes from health cache
>Sep 02 14:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:15:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:15:13.158756   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:15:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:15:13.159978   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:16:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:16:02.430962       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:16:37 Starting Node Health Status refresh
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:16:38 Cleaned 0 nodes from health cache
>Sep 02 14:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 18:17:11.413592044 +0000 UTC
>Sep 02 14:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:13.168Z] [INFO] Validating specs...
>Sep 02 14:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:13.168Z] [NOTICE] All specs passed validation!
>Sep 02 14:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:17:13.552Z] [INFO] Validating specs...
>Sep 02 14:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:13.552Z] [NOTICE] All specs passed validation!
>Sep 02 14:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:17:14.033Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:18:09.424452   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:18:13.251488   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:18:13.252376   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:18:37 Starting Node Health Status refresh
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:18:38 Cleaned 0 nodes from health cache
>Sep 02 14:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:19:34 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 103.105.49.148 replaced with 185.144.157.134
>Sep 02 14:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:20:37 Starting Node Health Status refresh
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:20:38 Cleaned 0 nodes from health cache
>Sep 02 14:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:21:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:21:13.254382   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:21:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:21:13.255357   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:21:58.007391       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:22:37 Starting Node Health Status refresh
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:22:38 Cleaned 0 nodes from health cache
>Sep 02 14:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:23:09.425792   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:24:12.182Z] [INFO] AnsibleBroker::Catalog
>Sep 02 14:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:18:24:12 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 14:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:24:13.336149   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:24:13.337084   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:24:37 Starting Node Health Status refresh
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:24:38 Cleaned 0 nodes from health cache
>Sep 02 14:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:26:37 Starting Node Health Status refresh
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:26:38 Cleaned 0 nodes from health cache
>Sep 02 14:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 18:27:11.41367671 +0000 UTC
>Sep 02 14:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:27:13.385296   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:27:13.386589   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:13.696Z] [INFO] Validating specs...
>Sep 02 14:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:13.696Z] [NOTICE] All specs passed validation!
>Sep 02 14:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:27:14.105Z] [INFO] Validating specs...
>Sep 02 14:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:14.105Z] [NOTICE] All specs passed validation!
>Sep 02 14:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:27:14.451Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:27:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:27:25.459530       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:28:09.427142   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:28:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:28:23.672632       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:28:37 Starting Node Health Status refresh
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:28:38 Cleaned 0 nodes from health cache
>Sep 02 14:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:30:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:30:13.467204   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:30:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:30:13.468404   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:30:37 Starting Node Health Status refresh
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:30:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:30:37 Cleaned 0 nodes from health cache
>Sep 02 14:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:31:38.024626       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:32:37 Starting Node Health Status refresh
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:32:38 Cleaned 0 nodes from health cache
>Sep 02 14:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com rhnsd[908]: /etc/sysconfig/rhn/systemid does not exist or is unreadable
>Sep 02 14:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:33:09.428637   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:33:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:33:13.473344   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:33:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:33:13.474291   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:34:37 Starting Node Health Status refresh
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:34:38 Cleaned 0 nodes from health cache
>Sep 02 14:34:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:34:48.497916       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:36:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:36:13.548705   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:36:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:36:13.549828   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:36:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:36:25.774819       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:36:37 Starting Node Health Status refresh
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:36:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:36:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:36:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:36:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:36:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:36:39 Cleaned 0 nodes from health cache
>Sep 02 14:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 18:37:11.41413037 +0000 UTC
>Sep 02 14:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:13.479Z] [INFO] Validating specs...
>Sep 02 14:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:13.479Z] [NOTICE] All specs passed validation!
>Sep 02 14:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:37:13.878Z] [INFO] Validating specs...
>Sep 02 14:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:13.879Z] [NOTICE] All specs passed validation!
>Sep 02 14:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:37:14.146Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:38:09.431348   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:38:37 Starting Node Health Status refresh
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:38:38 Cleaned 0 nodes from health cache
>Sep 02 14:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:39:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:39:13.597221   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:39:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:39:13.598403   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:40:37 Starting Node Health Status refresh
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:40:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:40:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:40:38 Cleaned 0 nodes from health cache
>Sep 02 14:40:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:40:56.262Z] [INFO] AnsibleBroker::Catalog
>Sep 02 14:40:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:18:40:56 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 14:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:41:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:41:12.099253       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:42:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:42:13.662519   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:42:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:42:13.664048   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:42:37 Starting Node Health Status refresh
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:42:38 Cleaned 0 nodes from health cache
>Sep 02 14:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:43:09.432943   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:44:37 Starting Node Health Status refresh
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:44:38 Cleaned 0 nodes from health cache
>Sep 02 14:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:45:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:45:13.717294   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:45:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:45:13.718278   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:46:37 Starting Node Health Status refresh
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:46:38 Cleaned 0 nodes from health cache
>Sep 02 14:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 18:47:11.413579637 +0000 UTC
>Sep 02 14:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:13.651Z] [INFO] Validating specs...
>Sep 02 14:47:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:13.651Z] [NOTICE] All specs passed validation!
>Sep 02 14:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:47:14.058Z] [INFO] Validating specs...
>Sep 02 14:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:14.058Z] [NOTICE] All specs passed validation!
>Sep 02 14:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:47:14.321Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:48:09.434361   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:48:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:48:13.754948   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:48:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:48:13.755898   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:48:37 Starting Node Health Status refresh
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:48:38 Cleaned 0 nodes from health cache
>Sep 02 14:48:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:48:54.898324       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:48:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:48:56.201343       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:50:37 Starting Node Health Status refresh
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:50:38 Cleaned 0 nodes from health cache
>Sep 02 14:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:51:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:51:13.853309   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:51:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:51:13.854155   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:51:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:51:20.591051       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:52:37 Starting Node Health Status refresh
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:52:38 Cleaned 0 nodes from health cache
>Sep 02 14:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:53:09.435928   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:54:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:54:13.941918   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:54:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:54:13.943988   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:54:37 Starting Node Health Status refresh
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:54:38 Cleaned 0 nodes from health cache
>Sep 02 14:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:56:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:56:19.333Z] [INFO] AnsibleBroker::Catalog
>Sep 02 14:56:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:18:56:19 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 14:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:56:37 Starting Node Health Status refresh
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:56:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:56:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:56:38 Cleaned 0 nodes from health cache
>Sep 02 14:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 14:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 18:57:11.413582097 +0000 UTC
>Sep 02 14:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 14:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:13.564Z] [INFO] Validating specs...
>Sep 02 14:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:13.564Z] [NOTICE] All specs passed validation!
>Sep 02 14:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:57:13.961994   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 14:57:13.963972   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 14:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T18:57:14.009Z] [INFO] Validating specs...
>Sep 02 14:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:14.009Z] [NOTICE] All specs passed validation!
>Sep 02 14:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T18:57:14.347Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 14:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 14:58:09.437376   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 14:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:58:37 Starting Node Health Status refresh
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 14:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 18:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 18:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 14:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 18:58:38 Cleaned 0 nodes from health cache
>Sep 02 14:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 14:59:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 18:59:10.704757       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 14:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 14:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 14:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 14:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 14:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:00:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:00:14.059589   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:00:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:00:14.060663   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 15:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 15:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:00:37 Starting Node Health Status refresh
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:00:38 Cleaned 0 nodes from health cache
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 61 of user root.
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 61 of user root.
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[15354]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[15357]: starting 0anacron
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[15363]: finished 0anacron
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 15:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 15:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:02:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:02:01.316478       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:02:08 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 173.71.68.101 replaced with 64.52.22.123
>Sep 02 15:02:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:02:19.978565       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:02:37 Starting Node Health Status refresh
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:02:38 Cleaned 0 nodes from health cache
>Sep 02 15:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:03:09.441476   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:03:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:03:14.152928   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:03:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:03:14.153886   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:04:37 Starting Node Health Status refresh
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:04:38 Cleaned 0 nodes from health cache
>Sep 02 15:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:06:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:06:14.242731   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:06:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:06:14.243691   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:06:37 Starting Node Health Status refresh
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:06:38 Cleaned 0 nodes from health cache
>Sep 02 15:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:07:11.413583453 +0000 UTC
>Sep 02 15:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:13.481Z] [INFO] Validating specs...
>Sep 02 15:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:13.482Z] [NOTICE] All specs passed validation!
>Sep 02 15:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:07:13.867Z] [INFO] Validating specs...
>Sep 02 15:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:13.867Z] [NOTICE] All specs passed validation!
>Sep 02 15:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:07:14.263Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:08:09.445966   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:08:37 Starting Node Health Status refresh
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:08:38 Cleaned 0 nodes from health cache
>Sep 02 15:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:09:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:09:14.280850   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:09:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:09:14.281763   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:10:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:10:27.010017       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:10:37 Starting Node Health Status refresh
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:10:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:10:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:10:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:10:39 Cleaned 0 nodes from health cache
>Sep 02 15:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:11:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:11:45.450303       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:12:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:12:14.381010   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:12:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:12:14.382028   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:12:37 Starting Node Health Status refresh
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:12:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:12:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:12:38 Cleaned 0 nodes from health cache
>Sep 02 15:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:13:09.447790   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:14:22.464Z] [INFO] AnsibleBroker::Catalog
>Sep 02 15:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:19:14:22 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 15:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:14:37 Starting Node Health Status refresh
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:14:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:14:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:14:38 Cleaned 0 nodes from health cache
>Sep 02 15:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:15:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:15:14.466089   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:15:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:15:14.467269   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:15:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:15:39.828365       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:16:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:16:37 Starting Node Health Status refresh
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:16:38 Cleaned 0 nodes from health cache
>Sep 02 15:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:17:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:17:11.413604444 +0000 UTC
>Sep 02 15:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:13.427Z] [INFO] Validating specs...
>Sep 02 15:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:13.427Z] [NOTICE] All specs passed validation!
>Sep 02 15:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:17:13.825Z] [INFO] Validating specs...
>Sep 02 15:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:13.825Z] [NOTICE] All specs passed validation!
>Sep 02 15:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:17:14.427Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:18:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:18:09.449172   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:18:14.536509   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:18:14.537497   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:18:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:18:37 Starting Node Health Status refresh
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:18:38 Cleaned 0 nodes from health cache
>Sep 02 15:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:19:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:19:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:20:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:20:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:20:37 Starting Node Health Status refresh
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:20:38 Cleaned 0 nodes from health cache
>Sep 02 15:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:21:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:21:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:21:14.583110   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:21:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:21:14.584298   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:21:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:22:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:22:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:22:37 Starting Node Health Status refresh
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:22:37 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:22:37 Cleaned 0 nodes from health cache
>Sep 02 15:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:23:09.450791   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:23:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:24:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:24:14.646152   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:24:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:24:14.646979   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:24:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:24:17.045987       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:24:37 Starting Node Health Status refresh
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:24:38 Cleaned 0 nodes from health cache
>Sep 02 15:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:26:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:26:16.550322       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:26:37 Starting Node Health Status refresh
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:26:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:26:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:26:38 Cleaned 0 nodes from health cache
>Sep 02 15:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:27:11.413557664 +0000 UTC
>Sep 02 15:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:13.899Z] [INFO] Validating specs...
>Sep 02 15:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:13.899Z] [NOTICE] All specs passed validation!
>Sep 02 15:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:27:14.324Z] [INFO] Validating specs...
>Sep 02 15:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:14.324Z] [NOTICE] All specs passed validation!
>Sep 02 15:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:27:14.668219   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:27:14.669279   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:27:15.181Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:28:09.452541   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:28:37 Starting Node Health Status refresh
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:28:38 Cleaned 0 nodes from health cache
>Sep 02 15:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:30:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:30:00.946867       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:30:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:30:11.605Z] [INFO] AnsibleBroker::Catalog
>Sep 02 15:30:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:19:30:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 15:30:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:30:14.671739   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:30:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:30:14.673330   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:30:37 Starting Node Health Status refresh
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:30:38 Cleaned 0 nodes from health cache
>Sep 02 15:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:32:37 Starting Node Health Status refresh
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:32:38 Cleaned 0 nodes from health cache
>Sep 02 15:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:33:09.454753   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:33:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:33:14.732906   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:33:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:33:14.733725   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:34:37 Starting Node Health Status refresh
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:34:38 Cleaned 0 nodes from health cache
>Sep 02 15:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:36:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:36:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:36:11.159501       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:36:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:36:14.780561   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:36:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:36:14.781539   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:36:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:36:37 Starting Node Health Status refresh
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:36:38 Cleaned 0 nodes from health cache
>Sep 02 15:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:37:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:37:11.413625423 +0000 UTC
>Sep 02 15:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:13.45Z] [INFO] Validating specs...
>Sep 02 15:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:13.45Z] [NOTICE] All specs passed validation!
>Sep 02 15:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:37:13.827Z] [INFO] Validating specs...
>Sep 02 15:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:13.836Z] [NOTICE] All specs passed validation!
>Sep 02 15:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:37:14.877Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:37:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:38:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:38:09.456283   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:38:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:38:37 Starting Node Health Status refresh
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:38:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:38:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:38:38 Cleaned 0 nodes from health cache
>Sep 02 15:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:39:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:39:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:39:14.857396   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:39:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:39:14.858280   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:39:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:40:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:40:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:40:37 Starting Node Health Status refresh
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:40:38 Cleaned 0 nodes from health cache
>Sep 02 15:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:41:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:41:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:42:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:42:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:42:14.878835   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:42:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:42:14.879861   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:42:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:42:17.667077       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:42:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:42:37 Starting Node Health Status refresh
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:42:38 Cleaned 0 nodes from health cache
>Sep 02 15:42:54 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 184.105.182.15 replaced with 204.44.81.178
>Sep 02 15:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:43:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:43:09.457912   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:43:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:44:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:44:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:44:37 Starting Node Health Status refresh
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:44:38 Cleaned 0 nodes from health cache
>Sep 02 15:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:45:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:45:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:45:10.987689       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:45:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:45:14.962891   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:45:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:45:14.963953   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:45:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:46:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:46:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:46:37 Starting Node Health Status refresh
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:46:38 Cleaned 0 nodes from health cache
>Sep 02 15:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:47:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:47:11.413565087 +0000 UTC
>Sep 02 15:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:14.673Z] [INFO] Validating specs...
>Sep 02 15:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:14.674Z] [NOTICE] All specs passed validation!
>Sep 02 15:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:47:15.065Z] [INFO] Validating specs...
>Sep 02 15:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:15.066Z] [NOTICE] All specs passed validation!
>Sep 02 15:47:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:47:16.039Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:47:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:48:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:48:09.459173   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:48:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:48:15.052569   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:48:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:48:15.053461   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:48:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:48:37 Starting Node Health Status refresh
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:48:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:48:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:48:38 Cleaned 0 nodes from health cache
>Sep 02 15:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:49:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:49:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:50:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:50:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:50:11.607Z] [INFO] AnsibleBroker::Catalog
>Sep 02 15:50:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:19:50:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 15:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:50:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:50:37 Starting Node Health Status refresh
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:50:38 Cleaned 0 nodes from health cache
>Sep 02 15:50:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:50:58.199293       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:51:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:51:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:51:15.088881   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:51:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:51:15.089723   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:51:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:52:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:52:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:52:10.746910       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:52:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:52:37 Starting Node Health Status refresh
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:52:38 Cleaned 0 nodes from health cache
>Sep 02 15:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:53:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:53:09.460710   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:53:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:54:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:54:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:54:15.103303   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:54:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:54:15.104342   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:54:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:54:37 Starting Node Health Status refresh
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:54:38 Cleaned 0 nodes from health cache
>Sep 02 15:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:55:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:55:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:56:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:56:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:56:37 Starting Node Health Status refresh
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:56:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:56:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:56:38 Cleaned 0 nodes from health cache
>Sep 02 15:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:57:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 15:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 19:57:11.413574766 +0000 UTC
>Sep 02 15:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 15:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:13.828Z] [INFO] Validating specs...
>Sep 02 15:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:13.828Z] [NOTICE] All specs passed validation!
>Sep 02 15:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T19:57:14.224Z] [INFO] Validating specs...
>Sep 02 15:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:14.224Z] [NOTICE] All specs passed validation!
>Sep 02 15:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T19:57:14.591Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 15:57:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:57:15.167855   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:57:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 15:57:15.168899   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 15:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:57:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:58:05.052173       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:58:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 15:58:09.462069   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 15:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:58:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:58:37 Starting Node Health Status refresh
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 15:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 19:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 19:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 15:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 19:58:38 Cleaned 0 nodes from health cache
>Sep 02 15:59:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 19:59:04.764184       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 15:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 15:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 15:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 15:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 15:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 15:59:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:00:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:00:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:00:15.267331   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:00:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:00:15.268583   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 16:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 16:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:00:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:00:37 Starting Node Health Status refresh
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:00:38 Cleaned 0 nodes from health cache
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 62 of user root.
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[21541]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 62 of user root.
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[21544]: starting 0anacron
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[21550]: finished 0anacron
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 16:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 16:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:01:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:01:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:02:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:02:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:02:37 Starting Node Health Status refresh
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:02:38 Cleaned 0 nodes from health cache
>Sep 02 16:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:03:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:03:09.463435   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:03:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:03:15.288593   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:03:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:03:15.289486   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:03:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:04:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:04:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:04:37 Starting Node Health Status refresh
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:04:38 Cleaned 0 nodes from health cache
>Sep 02 16:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:05:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:05:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:06:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:06:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:06:15.352149   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:06:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:06:15.353170   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:06:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:06:37 Starting Node Health Status refresh
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:06:38 Cleaned 0 nodes from health cache
>Sep 02 16:06:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:06:41.309413       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:07:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:11.415Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:11.415Z] [INFO] Attempting bootstrap at 2018-09-02 20:07:11.415050405 +0000 UTC
>Sep 02 16:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:11.415Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:13.44Z] [INFO] Validating specs...
>Sep 02 16:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:13.441Z] [NOTICE] All specs passed validation!
>Sep 02 16:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:07:13.857Z] [INFO] Validating specs...
>Sep 02 16:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:13.857Z] [NOTICE] All specs passed validation!
>Sep 02 16:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:07:14.43Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:07:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:08:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:08:09.465981   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:08:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:08:37 Starting Node Health Status refresh
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:08:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:08:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:08:38 Cleaned 0 nodes from health cache
>Sep 02 16:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:09:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:09:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:09:15.450924   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:09:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:09:15.451854   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:09:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:10:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:10:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:10:11.612Z] [INFO] AnsibleBroker::Catalog
>Sep 02 16:10:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:20:10:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 16:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:10:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:10:37 Starting Node Health Status refresh
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:10:38 Cleaned 0 nodes from health cache
>Sep 02 16:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:11:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:11:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:12:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:12:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:12:15.489210   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:12:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:12:15.490275   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:12:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:12:37 Starting Node Health Status refresh
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:12:38 Cleaned 0 nodes from health cache
>Sep 02 16:12:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:12:55.856648       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:13:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:13:09.467518   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:13:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:13:22.108729       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:13:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:14:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:14:37 Starting Node Health Status refresh
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:14:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:14:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:14:38 Cleaned 0 nodes from health cache
>Sep 02 16:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:15:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:15:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:15:15.497900   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:15:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:15:15.499003   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:15:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:15:25.324681       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:15:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:16:37 Starting Node Health Status refresh
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:16:38 Cleaned 0 nodes from health cache
>Sep 02 16:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 20:17:11.413684623 +0000 UTC
>Sep 02 16:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:15.988Z] [INFO] Validating specs...
>Sep 02 16:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:15.988Z] [NOTICE] All specs passed validation!
>Sep 02 16:17:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:17:16.362Z] [INFO] Validating specs...
>Sep 02 16:17:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:16.363Z] [NOTICE] All specs passed validation!
>Sep 02 16:17:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:17:16.893Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:18:09.469090   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:18:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:18:15.573929   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:18:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:18:15.574938   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:18:37 Starting Node Health Status refresh
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:18:38 Cleaned 0 nodes from health cache
>Sep 02 16:18:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:18:48.903458       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:20:37 Starting Node Health Status refresh
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:20:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:20:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:20:38 Cleaned 0 nodes from health cache
>Sep 02 16:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:21:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:21:15.675845   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:21:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:21:15.676941   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 72.5.72.15 replaced with 192.73.242.152
>Sep 02 16:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:22:37 Starting Node Health Status refresh
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:22:38.125159       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:22:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:22:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:22:39 Cleaned 0 nodes from health cache
>Sep 02 16:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:23:09.470601   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:24:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:24:15.774138   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:24:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:24:15.775046   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:24:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:24:24.346301       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:24:37 Starting Node Health Status refresh
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:24:38 Cleaned 0 nodes from health cache
>Sep 02 16:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:26:37 Starting Node Health Status refresh
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:26:38 Cleaned 0 nodes from health cache
>Sep 02 16:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 20:27:11.414211437 +0000 UTC
>Sep 02 16:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:13.334Z] [INFO] Validating specs...
>Sep 02 16:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:13.335Z] [NOTICE] All specs passed validation!
>Sep 02 16:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:27:13.749Z] [INFO] Validating specs...
>Sep 02 16:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:13.75Z] [NOTICE] All specs passed validation!
>Sep 02 16:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:14.319Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:27:15.854430   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:27:15.855613   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:27:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:27:49.806Z] [INFO] AnsibleBroker::Catalog
>Sep 02 16:27:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:20:27:49 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 16:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:28:09.472261   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:28:37 Starting Node Health Status refresh
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:28:38 Cleaned 0 nodes from health cache
>Sep 02 16:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:30:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:30:15.956136   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:30:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:30:15.957098   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:30:37 Starting Node Health Status refresh
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:30:38 Cleaned 0 nodes from health cache
>Sep 02 16:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:31:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:31:49.236136       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:32:37 Starting Node Health Status refresh
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:32:38 Cleaned 0 nodes from health cache
>Sep 02 16:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:33:09.473549   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:33:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:33:15.970700   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:33:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:33:15.971624   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:33:55 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:33:55.023636       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:34:37 Starting Node Health Status refresh
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:34:38 Cleaned 0 nodes from health cache
>Sep 02 16:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:36:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:36:15.992008   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:36:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:36:15.993923   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:36:37 Starting Node Health Status refresh
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:36:38 Cleaned 0 nodes from health cache
>Sep 02 16:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 20:37:11.414051534 +0000 UTC
>Sep 02 16:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:13.386Z] [INFO] Validating specs...
>Sep 02 16:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:13.386Z] [NOTICE] All specs passed validation!
>Sep 02 16:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:37:13.806Z] [INFO] Validating specs...
>Sep 02 16:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:13.813Z] [NOTICE] All specs passed validation!
>Sep 02 16:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:37:14.143Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:38:09.476734   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:38:37 Starting Node Health Status refresh
>Sep 02 16:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:38:38 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:38:38 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:38:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:38:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:38:38 Cleaned 0 nodes from health cache
>Sep 02 16:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:39:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:39:16.063391   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:39:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:39:16.064386   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:39:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:39:39.371128       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:40:37 Starting Node Health Status refresh
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:40:38 Cleaned 0 nodes from health cache
>Sep 02 16:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:42:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:42:16.156286   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:42:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:42:16.157392   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:42:37 Starting Node Health Status refresh
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:42:38 Cleaned 0 nodes from health cache
>Sep 02 16:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:43:09.478914   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:44:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:44:23.310062       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:44:37 Starting Node Health Status refresh
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:44:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:44:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:44:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:44:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:44:38 Cleaned 0 nodes from health cache
>Sep 02 16:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:45:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:45:16.184076   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:45:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:45:16.185126   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:46:37 Starting Node Health Status refresh
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:46:38 Cleaned 0 nodes from health cache
>Sep 02 16:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 20:47:11.41431248 +0000 UTC
>Sep 02 16:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:14.496Z] [INFO] Validating specs...
>Sep 02 16:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:14.496Z] [NOTICE] All specs passed validation!
>Sep 02 16:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:47:14.911Z] [INFO] Validating specs...
>Sep 02 16:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:14.912Z] [NOTICE] All specs passed validation!
>Sep 02 16:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:15.582Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:47:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:47:21.905Z] [INFO] AnsibleBroker::Catalog
>Sep 02 16:47:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:20:47:21 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 16:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:48:09.480452   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:48:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:48:16.261544   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:48:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:48:16.262482   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:48:37 Starting Node Health Status refresh
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:48:38 Cleaned 0 nodes from health cache
>Sep 02 16:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:49:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:49:07.440537       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:49:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:49:59.135555       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:50:37 Starting Node Health Status refresh
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:50:38 Cleaned 0 nodes from health cache
>Sep 02 16:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:51:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:51:16.278621   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:51:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:51:16.280020   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:52:37 Starting Node Health Status refresh
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:52:38 Cleaned 0 nodes from health cache
>Sep 02 16:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:53:09.481659   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:54:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:54:16.372209   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:54:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:54:16.373202   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:54:37 Starting Node Health Status refresh
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:54:38 Cleaned 0 nodes from health cache
>Sep 02 16:54:59 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 185.144.157.134 replaced with 103.105.51.156
>Sep 02 16:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:56:37 Starting Node Health Status refresh
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:56:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:56:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:56:38 Cleaned 0 nodes from health cache
>Sep 02 16:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:11.414Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 16:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 20:57:11.414373395 +0000 UTC
>Sep 02 16:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 16:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:13.663Z] [INFO] Validating specs...
>Sep 02 16:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:13.663Z] [NOTICE] All specs passed validation!
>Sep 02 16:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T20:57:14.076Z] [INFO] Validating specs...
>Sep 02 16:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:14.076Z] [NOTICE] All specs passed validation!
>Sep 02 16:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T20:57:14.508Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 16:57:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:57:16.472572   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:57:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 16:57:16.473640   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 16:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 16:58:09.482814   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 16:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:58:37 Starting Node Health Status refresh
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 16:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 20:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 20:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 16:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 20:58:38 Cleaned 0 nodes from health cache
>Sep 02 16:59:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:59:05.182354       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 16:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 16:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 16:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 16:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 16:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 16:59:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 20:59:40.343134       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:00:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:00:16.572336   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:00:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:00:16.573353   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 17:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 17:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:00:37 Starting Node Health Status refresh
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:00:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:00:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:00:38 Cleaned 0 nodes from health cache
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 63 of user root.
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 63 of user root.
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[27423]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[27426]: starting 0anacron
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[27432]: finished 0anacron
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 17:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 17:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:02:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:02:30.560579       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:02:37 Starting Node Health Status refresh
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:02:38 Cleaned 0 nodes from health cache
>Sep 02 17:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:03:09.484718   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:03:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:03:16.649393   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:03:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:03:16.650795   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:04:37 Starting Node Health Status refresh
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:04:38 Cleaned 0 nodes from health cache
>Sep 02 17:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:06:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:06:16.679848   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:06:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:06:16.680881   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:06:37 Starting Node Health Status refresh
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:06:38 Cleaned 0 nodes from health cache
>Sep 02 17:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:04.047Z] [INFO] AnsibleBroker::Catalog
>Sep 02 17:07:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:21:07:04 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 17:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:07:11.413609988 +0000 UTC
>Sep 02 17:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:14.862Z] [INFO] Validating specs...
>Sep 02 17:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:14.862Z] [NOTICE] All specs passed validation!
>Sep 02 17:07:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:07:15.304Z] [INFO] Validating specs...
>Sep 02 17:07:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:15.313Z] [NOTICE] All specs passed validation!
>Sep 02 17:07:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:07:16.168Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:08:09.488222   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:08:37 Starting Node Health Status refresh
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:08:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:08:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:08:38 Cleaned 0 nodes from health cache
>Sep 02 17:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:09:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:09:16.759765   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:09:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:09:16.760874   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:10:37 Starting Node Health Status refresh
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:10:38 Cleaned 0 nodes from health cache
>Sep 02 17:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:12:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:12:16.815412   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:12:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:12:16.816420   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:12:37 Starting Node Health Status refresh
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:12:38 Cleaned 0 nodes from health cache
>Sep 02 17:12:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:12:49.433150       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:13:09.489818   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:14:37 Starting Node Health Status refresh
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:14:38 Cleaned 0 nodes from health cache
>Sep 02 17:15:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:15:04.276339       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:15:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:15:16.866456   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:15:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:15:16.867396   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:16:06.589750       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:16:37 Starting Node Health Status refresh
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:16:38 Cleaned 0 nodes from health cache
>Sep 02 17:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:17:11.413641365 +0000 UTC
>Sep 02 17:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:13.385Z] [INFO] Validating specs...
>Sep 02 17:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:13.386Z] [NOTICE] All specs passed validation!
>Sep 02 17:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:17:13.803Z] [INFO] Validating specs...
>Sep 02 17:17:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:13.803Z] [NOTICE] All specs passed validation!
>Sep 02 17:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:17:14.294Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:18:09.491115   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:18:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:18:16.964389   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:18:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:18:16.965467   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:18:37 Starting Node Health Status refresh
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:18:38 Cleaned 0 nodes from health cache
>Sep 02 17:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:20:37 Starting Node Health Status refresh
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:20:38 Cleaned 0 nodes from health cache
>Sep 02 17:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:21:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:21:16.982385   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:21:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:21:16.983620   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:22:37 Starting Node Health Status refresh
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:22:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:22:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:22:39 Cleaned 0 nodes from health cache
>Sep 02 17:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:23:09.492614   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:24:08.691282       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:24:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:24:16.987682   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:24:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:24:16.988577   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:24:37 Starting Node Health Status refresh
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:24:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:24:38 Cleaned 0 nodes from health cache
>Sep 02 17:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:25:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:25:11.103Z] [INFO] AnsibleBroker::Catalog
>Sep 02 17:25:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:21:25:11 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 17:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:26:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:26:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:26:37 Starting Node Health Status refresh
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:26:38 Cleaned 0 nodes from health cache
>Sep 02 17:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:26:54.483697       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:27:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:27:11.413665036 +0000 UTC
>Sep 02 17:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:13.352Z] [INFO] Validating specs...
>Sep 02 17:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:13.353Z] [NOTICE] All specs passed validation!
>Sep 02 17:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:27:13.747Z] [INFO] Validating specs...
>Sep 02 17:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:13.747Z] [NOTICE] All specs passed validation!
>Sep 02 17:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:27:14.431Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:27:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:27:17.010036   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:27:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:27:17.011054   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:27:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:28:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:28:09.493885   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:28:33.373672       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:28:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:28:37 Starting Node Health Status refresh
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:28:38 Cleaned 0 nodes from health cache
>Sep 02 17:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:29:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:29:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:30:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:30:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:30:17.084808   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:30:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:30:17.085758   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:30:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:30:37 Starting Node Health Status refresh
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:30:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:30:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:30:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:30:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:30:39 Cleaned 0 nodes from health cache
>Sep 02 17:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:31:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:31:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:32:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:32:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:32:37 Starting Node Health Status refresh
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:32:38 Cleaned 0 nodes from health cache
>Sep 02 17:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:33:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:33:09.495079   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:33:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:33:17.181754   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:33:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:33:17.182531   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:33:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:34:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:34:37 Starting Node Health Status refresh
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:34:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:34:38 Cleaned 0 nodes from health cache
>Sep 02 17:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:35:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:35:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:36:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:36:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:36:17.184107   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:36:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:36:17.185152   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:36:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:36:37 Starting Node Health Status refresh
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:36:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:36:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:36:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:36:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:36:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:36:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:36:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:36:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:36:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:36:38 Cleaned 0 nodes from health cache
>Sep 02 17:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:37:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:37:11.4135803 +0000 UTC
>Sep 02 17:37:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:13.986Z] [INFO] Validating specs...
>Sep 02 17:37:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:13.986Z] [NOTICE] All specs passed validation!
>Sep 02 17:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:37:14.427Z] [INFO] Validating specs...
>Sep 02 17:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:14.427Z] [NOTICE] All specs passed validation!
>Sep 02 17:37:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:37:14.979Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:37:20 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 64.52.22.123 replaced with 69.195.159.158
>Sep 02 17:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:37:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:37:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:37:39.389204       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:38:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:38:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:38:09.496544   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:38:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:38:37 Starting Node Health Status refresh
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:38:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:38:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:38:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:38:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:38:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:38:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:38:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:38:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:38:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:38:38 Cleaned 0 nodes from health cache
>Sep 02 17:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:39:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:39:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:39:17.223096   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:39:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:39:17.224254   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:39:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:39:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:39:40.742455       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:40:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:40:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:40:37 Starting Node Health Status refresh
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:40:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:40:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:40:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:40:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:40:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:40:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:40:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:40:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:40:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:40:38 Cleaned 0 nodes from health cache
>Sep 02 17:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:41:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:41:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:41:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:41:52.597962       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:42:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:42:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:42:17.292092   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:42:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:42:17.293611   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:42:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:42:37 Starting Node Health Status refresh
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:42:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:42:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:42:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:42:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:42:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:42:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:42:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:42:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:42:38 Cleaned 0 nodes from health cache
>Sep 02 17:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:43:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:43:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:43:09.498099   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:43:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:44:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:44:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:44:21.151Z] [INFO] AnsibleBroker::Catalog
>Sep 02 17:44:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:21:44:21 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 17:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:44:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:44:37 Starting Node Health Status refresh
>Sep 02 17:44:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:44:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:44:38 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:44:38 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:44:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:44:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:44:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:44:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:44:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:44:38 Cleaned 0 nodes from health cache
>Sep 02 17:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:45:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:45:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:45:17.380560   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:45:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:45:17.381535   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:45:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:46:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:46:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:46:37 Starting Node Health Status refresh
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:46:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:46:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:46:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:46:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:46:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:46:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:46:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:46:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:46:38 Cleaned 0 nodes from health cache
>Sep 02 17:46:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:46:52.469396       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:47:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:47:11.413730069 +0000 UTC
>Sep 02 17:47:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:14.34Z] [INFO] Validating specs...
>Sep 02 17:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:14.34Z] [NOTICE] All specs passed validation!
>Sep 02 17:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:47:14.755Z] [INFO] Validating specs...
>Sep 02 17:47:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:14.755Z] [NOTICE] All specs passed validation!
>Sep 02 17:47:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:47:15.126Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:47:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:48:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:48:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:48:09.499346   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:48:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:48:17.393825   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:48:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:48:17.394734   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:48:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:48:37 Starting Node Health Status refresh
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:48:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:48:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:48:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:48:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:48:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:48:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:48:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:48:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:48:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:48:38 Cleaned 0 nodes from health cache
>Sep 02 17:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:49:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:49:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:50:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:50:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:50:13.694219       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:50:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:50:37 Starting Node Health Status refresh
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:50:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:50:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:50:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:50:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:50:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:50:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:50:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:50:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:50:38 Cleaned 0 nodes from health cache
>Sep 02 17:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:51:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:51:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:51:17.473688   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:51:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:51:17.474655   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:51:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:52:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:52:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:52:37 Starting Node Health Status refresh
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:52:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:52:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:52:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:52:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:52:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:52:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:52:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:52:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:52:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:52:38 Cleaned 0 nodes from health cache
>Sep 02 17:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:53:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:53:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:53:09.501046   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:53:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:53:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:53:39.664678   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:53:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:53:39.666538   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:54:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:54:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:54:17.477789   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:54:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:54:17.478840   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:54:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:54:19.660849   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:54:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:54:19.665562   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:54:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:54:37 Starting Node Health Status refresh
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:54:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:54:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:54:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:54:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:54:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:54:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:54:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:54:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:54:38 Cleaned 0 nodes from health cache
>Sep 02 17:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:55:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:55:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:55:19.660667   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:55:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:55:19.665637   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 17:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:55:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:56:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:56:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:56:37 Starting Node Health Status refresh
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:56:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:56:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:56:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:56:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:56:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:56:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:56:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:56:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:56:38 Cleaned 0 nodes from health cache
>Sep 02 17:56:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 21:56:44.793312       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 17:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:57:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 17:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 21:57:11.413570608 +0000 UTC
>Sep 02 17:57:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 17:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:13.358Z] [INFO] Validating specs...
>Sep 02 17:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:13.359Z] [NOTICE] All specs passed validation!
>Sep 02 17:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T21:57:13.755Z] [INFO] Validating specs...
>Sep 02 17:57:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:13.755Z] [NOTICE] All specs passed validation!
>Sep 02 17:57:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T21:57:14.282Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 17:57:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:57:17.568288   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:57:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 17:57:17.569498   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 17:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:57:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:58:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:58:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 17:58:09.502432   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 17:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:58:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:58:37 Starting Node Health Status refresh
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:58:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:58:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:58:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:58:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 17:58:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 21:58:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 21:58:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:58:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 17:58:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 21:58:38 Cleaned 0 nodes from health cache
>Sep 02 17:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:59:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 17:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 17:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 17:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 17:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 17:59:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:00:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:00:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:00:13.495376       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:00:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:00:17.586207   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:00:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:00:17.587201   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Docker Cleanup...
>Sep 02 18:00:18 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Docker Cleanup.
>Sep 02 18:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:00:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:00:37 Starting Node Health Status refresh
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:00:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:00:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:00:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:00:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:00:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:00:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:00:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:00:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:00:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:00:38 Cleaned 0 nodes from health cache
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of root.
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of root.
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 64 of user root.
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 64 of user root.
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com CROND[872]: (root) CMD (run-parts /etc/cron.hourly)
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[875]: starting 0anacron
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com run-parts(/etc/cron.hourly)[881]: finished 0anacron
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice User Slice of root.
>Sep 02 18:01:01 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping User Slice of root.
>Sep 02 18:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:01:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:01:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:02:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:02:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:02:37 Starting Node Health Status refresh
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:02:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:02:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:02:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:02:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:02:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:02:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:02:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:02:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:02:38 Cleaned 0 nodes from health cache
>Sep 02 18:02:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:02:49.808109       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:03:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:03:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:03:09.504080   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:03:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:03:17.672626   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:03:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:03:17.673481   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:03:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:03:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:03:43.606Z] [INFO] AnsibleBroker::Catalog
>Sep 02 18:03:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:22:03:43 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 18:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:04:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:04:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:04:37 Starting Node Health Status refresh
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:04:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:04:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:04:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:04:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:04:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:04:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:04:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:04:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:04:38 Cleaned 0 nodes from health cache
>Sep 02 18:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:05:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:05:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:06:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:06:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:06:17.736519   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:06:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:06:17.737618   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:06:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:06:37 Starting Node Health Status refresh
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:06:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:06:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:06:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:06:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:06:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:06:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:06:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:06:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:06:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:06:38 Cleaned 0 nodes from health cache
>Sep 02 18:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:07:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 18:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:11.413Z] [INFO] Attempting bootstrap at 2018-09-02 22:07:11.413658549 +0000 UTC
>Sep 02 18:07:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:11.413Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 18:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:13.748Z] [INFO] Validating specs...
>Sep 02 18:07:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:13.748Z] [NOTICE] All specs passed validation!
>Sep 02 18:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T22:07:14.136Z] [INFO] Validating specs...
>Sep 02 18:07:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:14.137Z] [NOTICE] All specs passed validation!
>Sep 02 18:07:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:07:15.249Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 18:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:07:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:08:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:08:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:08:09.505961   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:08:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:08:37 Starting Node Health Status refresh
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:08:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:08:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:08:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:08:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:08:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:08:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:08:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:08:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:08:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:08:38 Cleaned 0 nodes from health cache
>Sep 02 18:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:09:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:09:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:09:17.790904   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:09:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:09:17.791815   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:09:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:10:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:10:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:10:37 Starting Node Health Status refresh
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:10:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:10:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:10:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:10:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:10:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:10:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:10:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:10:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:10:38 Cleaned 0 nodes from health cache
>Sep 02 18:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:11:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:11:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:11:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:11:47.908525       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:11:56 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:11:56.624065       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:12:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:12:00.830533       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:12:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:12:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:12:17.867317   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:12:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:12:17.868603   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:12:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:12:37 Starting Node Health Status refresh
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:12:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4049 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:12:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:12:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:12:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:12:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:12:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:12:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ 879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ4026 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:12:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:12:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:12:38 Cleaned 0 nodes from health cache
>Sep 02 18:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:13:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:13:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:13:09.507495   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:13:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:14:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started POST /volumes
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 Loaded simple allocator
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 brick_num: 0
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:14:18.609179   16429 proxier.go:320] Adding new service port "test/fod-postgresql-test:5432-tcp" at 172.30.33.121:5432/TCP
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 106.616423ms
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:14:18 asynchttp.go:125: Started job c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 Started async operation: Create Volume
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 Creating brick 0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 Creating brick 4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:18 Creating brick 5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 71.116Âµs
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir -p /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir -p /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir -p /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:14:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 136.604Âµs
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_cca670ded0d3d9406bfe8b42c7d40246/tp_4ef68640ba06fbe39c3dd46c0f553791 -V 1048576K -n brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_4ef68640ba06fbe39c3dd46c0f553791" created.
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_5010d0805890e4e816974cdc078d00d9/tp_5226dc43f96b03f119f87e30d82a08fc -V 1048576K -n brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:14:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_5226dc43f96b03f119f87e30d82a08fc" created.
>Sep 02 18:14:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_c8714ec5934f2236d1e8df8af51d1233/tp_0fa53e085a568e3cf52242b0f3d97a7f -V 1048576K -n brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:14:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:14:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_0fa53e085a568e3cf52242b0f3d97a7f" created.
>Sep 02 18:14:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 211.889Âµs
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_4ef68640ba06fbe39c3dd46c0f553791 isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: awk "BEGIN {print \"/dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_4ef68640ba06fbe39c3dd46c0f553791 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 120.639Âµs
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_5226dc43f96b03f119f87e30d82a08fc isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: awk "BEGIN {print \"/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_5226dc43f96b03f119f87e30d82a08fc /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 143.568Âµs
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_4ef68640ba06fbe39c3dd46c0f553791 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791/brick
>Sep 02 18:14:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chown :2002 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791/brick
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_5226dc43f96b03f119f87e30d82a08fc /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_c8714ec5934f2236d1e8df8af51d1233-brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_c8714ec5934f2236d1e8df8af51d1233-brick_0fa53e085a568e3cf52242b0f3d97a7f isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chmod 2775 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791/brick
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc/brick
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: awk "BEGIN {print \"/dev/mapper/vg_c8714ec5934f2236d1e8df8af51d1233-brick_0fa53e085a568e3cf52242b0f3d97a7f /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chown :2002 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc/brick
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chmod 2775 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc/brick
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 118.406Âµs
>Sep 02 18:14:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 146.152Âµs
>Sep 02 18:14:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 113.382Âµs
>Sep 02 18:14:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 164.241Âµs
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_c8714ec5934f2236d1e8df8af51d1233-brick_0fa53e085a568e3cf52242b0f3d97a7f /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f/brick
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chown :2002 /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f/brick
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chmod 2775 /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f/brick
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:14:27 Creating volume vol_7d596038f9459b6b830d9bbf32e36ca8 replica 3
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 163.685Âµs
>Sep 02 18:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.959299ms
>Sep 02 18:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 96.821Âµs
>Sep 02 18:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 102.788Âµs
>Sep 02 18:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 167.716Âµs
>Sep 02 18:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 121.581Âµs
>Sep 02 18:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume create vol_7d596038f9459b6b830d9bbf32e36ca8 replica 3 10.10.94.39:/var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f/brick 10.10.94.22:/var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791/brick 10.10.94.28:/var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc/brick
>Sep 02 18:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume create: vol_7d596038f9459b6b830d9bbf32e36ca8: success: please start the volume to access data
>Sep 02 18:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 116.235Âµs
>Sep 02 18:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 168.793Âµs
>Sep 02 18:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 189.435Âµs
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 160.834Âµs
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:14:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:37 Starting Node Health Status refresh
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:14:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13986 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13987 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:14:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 224.438Âµs
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13937 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13938 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:14:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:14:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume start vol_7d596038f9459b6b830d9bbf32e36ca8
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume start: vol_7d596038f9459b6b830d9bbf32e36ca8: success
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:38 Create Volume succeeded
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:14:38 asynchttp.go:129: Completed job c085ca5761f1e1e63910679c915ad71b in 19.754157489s
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:14:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13227 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13228 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:14:38 Cleaned 0 nodes from health cache
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/c085ca5761f1e1e63910679c915ad71b
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 303 See Other in 1.413451ms
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /volumes/7d596038f9459b6b830d9bbf32e36ca8
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 6.550912ms
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /clusters/d247fd8083c3eeba705dcaf3af33fcca
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 331.48Âµs
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/88667d40e7f24c804d9bdb941400045c
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.860424ms
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/b8388f778784f9d9dde7941d916af81b
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 932.46Âµs
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/cec8b27ee42e56f5ed75b25460c01cdf
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 804.511Âµs
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:14:38.879901   16429 roundrobin.go:276] LoadBalancerRR: Setting endpoints for test/glusterfs-dynamic-fod-postgresql-test: to [10.10.94.22:1 10.10.94.28:1 10.10.94.39:1]
>Sep 02 18:14:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:14:38.934190   16429 proxier.go:320] Adding new service port "test/glusterfs-dynamic-fod-postgresql-test:" at 172.30.136.143:1/TCP
>Sep 02 18:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:15:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:15:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:15:17.885637   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:15:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:15:17.887203   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:15:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:16:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:16:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:22.840936   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:16:22 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-besteffort-pod898869a4_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:16:22 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-besteffort-pod898869a4_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:16:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:22.917346   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "898869a4-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:23.018212   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "898869a4-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/898869a4-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/898869a4-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:23.052850   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "898869a4-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:23.223837   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:16:23 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:16:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:16:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576.
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576.
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[2521]: systemdhook <debug>: 26780b70f7f8: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:16:25 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[2522]: umounthook <debug>: prestart container_id:26780b70f7f8 rootfs:/var/lib/docker/devicemapper/mnt/62229a3baa87047ee80c7c133bf5b70086e87a1dc7514259c4b9fe2efe85e8a3/rootfs
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926586.1248] manager: (veth7b8586ef): new Veth device (/org/freedesktop/NetworkManager/Devices/17)
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926586.1377] device (veth7b8586ef): carrier: link connected
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:26.166348   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"898869a4-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576"}
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[2546]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth7b8586ef
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926586.1857] device (veth7b8586ef): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth7b8586ef entered promiscuous mode
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:16:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:16:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:16:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:16:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:16:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:16:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401.
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401.
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[2629]: systemdhook <debug>: 0bae69c55f42: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[2630]: umounthook <debug>: prestart container_id:0bae69c55f42 rootfs:/var/lib/docker/devicemapper/mnt/b770ebe00272013f685cdf09ddf92f65c5806fdb1cc0db43750d672a98c8be84/rootfs
>Sep 02 18:16:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:28.319765   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"898869a4-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401"}
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Scaling fod-postgresql-test-1 to 1
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.348008   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-burstable-podd88700d3_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-burstable-podd88700d3_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.489728   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.490878   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.601758   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:94e728bf-aefd-11e8-932d-fa163edaf27e,ResourceVersion:500184,Generation:0,CreationTimestamp:2018-09-02 18:14:38 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.605549   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.605628   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d88700d3-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d88700d3-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:16:32.634682   16429 container.go:354] Failed to create summary reader for "/system.slice/run-2644.scope": none of the resources are being tracked.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d88700d3-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d88700d3-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e.
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.655602   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.716156   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:94e728bf-aefd-11e8-932d-fa163edaf27e,ResourceVersion:500184,Generation:0,CreationTimestamp:2018-09-02 18:14:38 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.825674   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:94e728bf-aefd-11e8-932d-fa163edaf27e,ResourceVersion:500184,Generation:0,CreationTimestamp:2018-09-02 18:14:38 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.852430   16429 glusterfs.go:341] glusterfs: successfully mounted /var/lib/origin/openshift.local.volumes/pods/d88700d3-aefd-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e
>Sep 02 18:16:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:32.852507   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-rhjfk" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:33.010354   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:16:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49.
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49.
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[2836]: systemdhook <debug>: d8d29e652290: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:16:34 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[2837]: umounthook <debug>: prestart container_id:d8d29e652290 rootfs:/var/lib/docker/devicemapper/mnt/323d1134f800a70f0a58e157c790f79755b2820623ba0a5cc97c1bf8de935f36/rootfs
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:35.152356   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49"}
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926595.2085] manager: (veth7fea0021): new Veth device (/org/freedesktop/NetworkManager/Devices/18)
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926595.2134] device (veth7fea0021): carrier: link connected
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[2863]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth7fea0021
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926595.2398] device (veth7fea0021): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth7fea0021 entered promiscuous mode
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:16:35.313185060-04:00" level=error msg="Handler for GET /v1.24/images/registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03/json?size=1 returned error: No such image: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03"
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:16:35.313261058-04:00" level=error msg="Handler for GET /v1.24/images/registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03/json returned error: No such image: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03"
>Sep 02 18:16:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:35.315470   16429 provider.go:119] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
>Sep 02 18:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:16:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:16:37 Starting Node Health Status refresh
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:16:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13987 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:16:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:16:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13938 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:16:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:16:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:16:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:16:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13228 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:16:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:16:38 Cleaned 0 nodes from health cache
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:16:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:16:45 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:16:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:45.322809   16429 kube_docker_client.go:328] Pulling image "registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03": "15bfd2c7a1c8: Extracting [==================================================>] 33.06 MB/33.06 MB"
>Sep 02 18:16:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:45.511203   16429 kube_docker_client.go:331] Stop pulling image "registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03": "Digest: sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03"
>Sep 02 18:16:45 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:16:46 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856.
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856.
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[3022]: systemdhook <debug>: 522c00ed773b: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[3023]: umounthook <debug>: prestart container_id:522c00ed773b rootfs:/var/lib/docker/devicemapper/mnt/6a82031a2bc81d52a7616f86f9d8e6cd39a5fd14382ec81a0328feb206dc4ca2/rootfs
>Sep 02 18:16:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:47.332812   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856"}
>Sep 02 18:16:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:16:48.203609   16429 conversion.go:110] Could not get instant cpu stats: different number of cpus
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The files belonging to this database system will be owned by user "postgres".
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: This user must also own the server process.
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The database cluster will be initialized with locale "en_US.utf8".
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default database encoding has accordingly been set to "UTF8".
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default text search configuration will be set to "english".
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Data page checksums are disabled.
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:16:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
>Sep 02 18:16:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating subdirectories ... ok
>Sep 02 18:16:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default max_connections ... 100
>Sep 02 18:16:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default shared_buffers ... 128MB
>Sep 02 18:16:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting dynamic shared memory implementation ... posix
>Sep 02 18:16:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:16:52.605747   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:16:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:16:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:16:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:16:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:16:54 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating configuration files ... ok
>Sep 02 18:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:02.418735   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:17:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:17:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating template1 database in /var/lib/pgsql/data/userdata/base/1 ... ok
>Sep 02 18:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:11.415Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 18:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:11.416Z] [INFO] Attempting bootstrap at 2018-09-02 22:17:11.415564098 +0000 UTC
>Sep 02 18:17:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:11.416Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:12.408730   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:17:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing pg_authid ... ok
>Sep 02 18:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:14.492Z] [INFO] Validating specs...
>Sep 02 18:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:14.492Z] [NOTICE] All specs passed validation!
>Sep 02 18:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T22:17:14.93Z] [INFO] Validating specs...
>Sep 02 18:17:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:14.931Z] [NOTICE] All specs passed validation!
>Sep 02 18:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing dependencies ... ok
>Sep 02 18:17:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:17:15.465Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 18:17:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating system views ... ok
>Sep 02 18:17:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: loading system objects' descriptions ... ok
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:22.350257   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:17:22.399533383-04:00" level=warning msg="containerd: unable to save 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856:5d23f4f6967f5e0f33f5df8b96c598113d8cdfca7ffa89368eee92c89f8fbb6c starttime: open /proc/3192/stat: no such file or directory"
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:17:22.400740644-04:00" level=info msg="containerd: 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856:5d23f4f6967f5e0f33f5df8b96c598113d8cdfca7ffa89368eee92c89f8fbb6c (pid 3192) has become an orphan, killing it"
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:22.404336   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:17:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating collations ... ok
>Sep 02 18:17:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating conversions ... ok
>Sep 02 18:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating dictionaries ... ok
>Sep 02 18:17:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: setting privileges on built-in objects ... ok
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:32.349973   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:32.427965   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:17:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating information schema ... ok
>Sep 02 18:17:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: loading PL/pgSQL server-side language ... ok
>Sep 02 18:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:17:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:42.349595   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:42.351284   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:42.404684   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:42.653499   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856"}) of pod fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:17:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:42.654298   16429 kuberuntime_container.go:566] Killing container "docker://522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856" with 30 second grace period
>Sep 02 18:17:49 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: vacuuming database template1 ... ok
>Sep 02 18:17:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:17:52.417605   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:17:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:17:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:17:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:17:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:01 node-0.alxcns2.lab.rdu2.cee.redhat.com chronyd[476]: Source 204.44.81.178 replaced with 216.75.56.132
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:02.436765   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: copying template1 to template0 ... ok
>Sep 02 18:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:18:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:18:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:09.509841   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:18:12.414534205-04:00" level=warning msg="containerd: unable to save 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856:b5e71074a9b910219330a9b2b99f5e172a55cf47e7b0753fd5ed63114714c626 starttime: open /proc/3421/stat: no such file or directory"
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:18:12.416335373-04:00" level=info msg="containerd: 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856:b5e71074a9b910219330a9b2b99f5e172a55cf47e7b0753fd5ed63114714c626 (pid 3421) has become an orphan, killing it"
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:12.421442   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:18:12.657633415-04:00" level=info msg="Container 522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856 failed to exit within 30 seconds of signal 15 - using the force"
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[3443]: systemdhook <debug>: 522c00ed773b: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[3444]: umounthook <debug>: 522c00ed773b: only runs in prestart stage, ignoring
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: copying template1 to postgres ... 
>Sep 02 18:18:12 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:13.686079   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"522c00ed773b9671d7d85b0f18eb521f1f4b3f9ac1f89950b6ccb7a42f62a856"}
>Sep 02 18:18:13 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47.
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47.
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[3522]: systemdhook <debug>: 9945edd9faa7: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[3523]: umounthook <debug>: prestart container_id:9945edd9faa7 rootfs:/var/lib/docker/devicemapper/mnt/4c711fb123153018b3ba4bd47ff8fa7e11c257e381bd35572c3b851f1bde5f52/rootfs
>Sep 02 18:18:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:14.998599   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47"}
>Sep 02 18:18:15 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:18:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:18:16 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:18:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:18:17.919265   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:18:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:18:17.919359   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:18:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:18:17.919464   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:18:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:22.440290   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:32.438456   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:18:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:18:37 Starting Node Health Status refresh
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:18:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13987 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:18:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:18:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13938 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:18:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:18:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:18:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:18:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13228 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:18:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:18:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:18:38 Cleaned 0 nodes from health cache
>Sep 02 18:18:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:42.403127   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:52.349897   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:18:52.415091   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:18:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:02.349431   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:02.350692   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:02.396943672-04:00" level=warning msg="containerd: unable to save 9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47:fbb14c596e2619419846f4a2cd59bd603d1f2002bf2b4313d2f86791b1f0ed66 starttime: open /proc/3762/stat: no such file or directory"
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:02.397947339-04:00" level=info msg="containerd: 9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47:fbb14c596e2619419846f4a2cd59bd603d1f2002bf2b4313d2f86791b1f0ed66 (pid 3762) has become an orphan, killing it"
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:02.401053   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:02.652589   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47"}) of pod fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:19:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:02.653406   16429 kuberuntime_container.go:566] Killing container "docker://9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47" with 30 second grace period
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:19:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:19:06.993636       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:19:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:12.401894   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ......................................................... done
>Sep 02 18:19:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:19:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:19:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:19:14 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[3868]: systemdhook <debug>: 9945edd9faa7: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:19:14 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[3869]: umounthook <debug>: 9945edd9faa7: only runs in prestart stage, ignoring
>Sep 02 18:19:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:19:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:14.426556   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:14 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:15.188464   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47"}
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:19:15 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64.
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64.
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[3949]: systemdhook <debug>: 60943ea7b9ff: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[3950]: umounthook <debug>: prestart container_id:60943ea7b9ff rootfs:/var/lib/docker/devicemapper/mnt/536a27a79330bd0e53d43a7c90c44f440fd026ea6302a5b5ae31a335cc62f039/rootfs
>Sep 02 18:19:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:16.569092   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64"}
>Sep 02 18:19:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:19:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:19:17 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:19:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:22.436509   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:32.401278   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:19:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:19:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:42.409968   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:51.090982   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:51.092125   16429 kuberuntime_container.go:566] Killing container "docker://60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64" with 30 second grace period
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Success
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4148]: systemdhook <debug>: 0bae69c55f42: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4149]: umounthook <debug>: 0bae69c55f42: only runs in prestart stage, ignoring
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:51.591819   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:51.592950   16429 kuberuntime_container.go:566] Killing container "docker://0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401" with 10 second grace period
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:51.595735576-04:00" level=error msg="Handler for POST /v1.24/containers/0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401/stop?t=10 returned error: Container 0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401 is already stopped"
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:51.595764979-04:00" level=error msg="Handler for POST /v1.24/containers/0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401/stop returned error: Container 0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401 is already stopped"
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[4172]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 veth7b8586ef
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth7b8586ef left promiscuous mode
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4192]: systemdhook <debug>: 26780b70f7f8: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:19:51 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4193]: umounthook <debug>: 26780b70f7f8: only runs in prestart stage, ignoring
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.188411   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"898869a4-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"0bae69c55f422a64a22470b20b17ad74a89a0f4bc715036d698df211f3b17401"}
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.189142   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"898869a4-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576"}
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:19:52.189700   16429 pod_container_deletor.go:77] Container "26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576" not found in pod's containers
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:52.197283342-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop?t=10 returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:52.198004073-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.290757   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "898869a4-aefd-11e8-932d-fa163edaf27e" (UID: "898869a4-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.303993   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls" (OuterVolumeSpecName: "deployer-token-85tls") pod "898869a4-aefd-11e8-932d-fa163edaf27e" (UID: "898869a4-aefd-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "deployer-token-85tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.349853   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.396801   16429 reconciler.go:290] Volume detached for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/898869a4-aefd-11e8-932d-fa163edaf27e-deployer-token-85tls") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:52.420929   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:19:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:19:53 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:53.224567205-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop?t=10 returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:19:53 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:19:53.224618339-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:19:53 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-besteffort-pod898869a4_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:19:53 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-besteffort-pod898869a4_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:19:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:57.998730   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:58.004373   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:19:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:19:58.004871   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-deploy_test(898869a4-aefd-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:02.349468   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.9:5432: getsockopt: connection refused
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:02.351684   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:02.432161227-04:00" level=warning msg="containerd: unable to save 60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64:d5eb7f27d1f47e7c371af12da0fdae2c8d994687678db738a8508dba8ae2336f starttime: open /proc/4250/stat: no such file or directory"
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:02.433436995-04:00" level=info msg="containerd: 60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64:d5eb7f27d1f47e7c371af12da0fdae2c8d994687678db738a8508dba8ae2336f (pid 4250) has become an orphan, killing it"
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:02.437023   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:20:02 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:06.243656221-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop?t=10 returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:06.244624668-04:00" level=error msg="Handler for POST /v1.24/containers/26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576/stop returned error: Container 26780b70f7f811322797230e46f2956bf1d72cc8840140668ff6a76dba40b576 is already stopped"
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:20:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ...................................................... done
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:12.422961   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4354]: systemdhook <debug>: 60943ea7b9ff: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4355]: umounthook <debug>: 60943ea7b9ff: only runs in prestart stage, ignoring
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[4376]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 veth7fea0021
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth7fea0021 left promiscuous mode
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4396]: systemdhook <debug>: d8d29e652290: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4397]: umounthook <debug>: d8d29e652290: only runs in prestart stage, ignoring
>Sep 02 18:20:12 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.479843   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64"}
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.480665   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d88700d3-aefd-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49"}
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:20:13.481138   16429 pod_container_deletor.go:77] Container "d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49" not found in pod's containers
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:13.493402624-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop?t=10 returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:13.494275333-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.563609   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "fod-postgresql-test" (UniqueName: "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "d88700d3-aefd-11e8-932d-fa163edaf27e" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.564313   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq") pod "d88700d3-aefd-11e8-932d-fa163edaf27e" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e")
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.577906   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq" (OuterVolumeSpecName: "default-token-brthq") pod "d88700d3-aefd-11e8-932d-fa163edaf27e" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "default-token-brthq". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.591732   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (OuterVolumeSpecName: "fod-postgresql-test") pod "d88700d3-aefd-11e8-932d-fa163edaf27e" (UID: "d88700d3-aefd-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e". PluginName "kubernetes.io/glusterfs", VolumeGidValue "2002"
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.664999   16429 reconciler.go:290] Volume detached for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d88700d3-aefd-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:20:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:13.665043   16429 reconciler.go:290] Volume detached for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d88700d3-aefd-11e8-932d-fa163edaf27e-default-token-brthq") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-burstable-podd88700d3_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-burstable-podd88700d3_aefd_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:14.498046547-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop?t=10 returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:20:14.498073676-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:14.535713   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:14.548727   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)"
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:14.549204   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-rhjfk_test(d88700d3-aefd-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:20:14.981402   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47 with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/9945edd9faa7fe46eec8c7931f9e717345e6f1e5abb199552b892a7912815f47â: No such file or directory
>Sep 02 18:20:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:20:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:20:16.534704   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64 with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/60943ea7b9ff53283c94d9c2faaea5739908e3bac2881a2f8fe94852b2278d64â: No such file or directory
>Sep 02 18:20:16 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:20:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:20:23.807833   16429 proxier.go:336] Removing service port "test/fod-postgresql-test:5432-tcp"
>Sep 02 18:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:20:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:20:37 Starting Node Health Status refresh
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:20:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13987 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:20:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:20:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13938 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:20:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:20:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:20:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:20:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13228 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:20:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:20:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:20:38 Cleaned 0 nodes from health cache
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:21:06.408188935-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop?t=10 returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:21:06.408247538-04:00" level=error msg="Handler for POST /v1.24/containers/d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49/stop returned error: Container d8d29e65229057662bc390f0d4e99c1f112489fdb30dde199c7578dd566abe49 is already stopped"
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:21:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:21:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:21:17.964449   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:21:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:21:17.965502   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.655189   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-besteffort-pod875ab363_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-besteffort-pod875ab363_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.688125   16429 proxier.go:320] Adding new service port "test/fod-postgresql-test:5432-tcp" at 172.30.80.150:5432/TCP
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.703958   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "875ab363-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.804982   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "875ab363-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/875ab363-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/875ab363-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:21:25.821117   16429 helpers.go:138] readString: Failed to read "/sys/fs/cgroup/memory/system.slice/run-4559.scope/memory.limit_in_bytes": open /sys/fs/cgroup/memory/system.slice/run-4559.scope/memory.limit_in_bytes: no such file or directory
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.830085   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "875ab363-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:25.994546   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:21:26 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:21:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23.
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23.
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4638]: systemdhook <debug>: 784f92f75c3d: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4639]: umounthook <debug>: prestart container_id:784f92f75c3d rootfs:/var/lib/docker/devicemapper/mnt/b1e0d26bcf9370421c0dd3fb486c844efdb171074e95529d68a78d6d877fdee3/rootfs
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:29.539374   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"875ab363-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23"}
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926889.6379] manager: (veth2fd639d7): new Veth device (/org/freedesktop/NetworkManager/Devices/19)
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926889.6508] device (veth2fd639d7): carrier: link connected
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[4663]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth2fd639d7
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926889.6727] device (veth2fd639d7): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth2fd639d7 entered promiscuous mode
>Sep 02 18:21:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:21:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:21:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d.
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d.
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:21:33 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4747]: systemdhook <debug>: 040a3fbca9c6: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4748]: umounthook <debug>: prestart container_id:040a3fbca9c6 rootfs:/var/lib/docker/devicemapper/mnt/f40189b13281abb967300d55aa0d44c17733292c3555faf315695b501e67ef67/rootfs
>Sep 02 18:21:34 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:34.310257   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"875ab363-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d"}
>Sep 02 18:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:21:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Scaling fod-postgresql-test-1 to 1
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.595096   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-burstable-pod8f105a78_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-burstable-pod8f105a78_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.666257   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.666930   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.767648   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/8f105a78-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/8f105a78-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.790424   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:94e728bf-aefd-11e8-932d-fa163edaf27e,ResourceVersion:500184,Generation:0,CreationTimestamp:2018-09-02 18:14:38 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.793989   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.798806   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/8f105a78-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/8f105a78-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e.
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.905413   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:94e728bf-aefd-11e8-932d-fa163edaf27e,ResourceVersion:500184,Generation:0,CreationTimestamp:2018-09-02 18:14:38 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.964507   16429 glusterfs.go:341] glusterfs: successfully mounted /var/lib/origin/openshift.local.volumes/pods/8f105a78-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e
>Sep 02 18:21:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:38.964642   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-bdrdv" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:21:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:39.222457   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:21:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:21:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:21:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:21:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a.
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a.
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[4979]: systemdhook <debug>: cef679970fe6: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:21:40 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[4980]: umounthook <debug>: prestart container_id:cef679970fe6 rootfs:/var/lib/docker/devicemapper/mnt/1b0b24734c267920253414ed8ed10174d607e624fec06ee1d4521451d33e04ea/rootfs
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:41.142459   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"8f105a78-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a"}
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926901.1621] manager: (veth03551a2b): new Veth device (/org/freedesktop/NetworkManager/Devices/20)
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926901.1718] device (veth03551a2b): carrier: link connected
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[5006]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth03551a2b
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535926901.2016] device (veth03551a2b): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth03551a2b entered promiscuous mode
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:21:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663.
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663.
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5086]: systemdhook <debug>: 463bd24bdbfd: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:21:42 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5087]: umounthook <debug>: prestart container_id:463bd24bdbfd rootfs:/var/lib/docker/devicemapper/mnt/7ac912a839778595f109b5bb4575068622fab025c6ab53abb086c86c74929e13/rootfs
>Sep 02 18:21:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:43.186922   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"8f105a78-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663"}
>Sep 02 18:21:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:48.692300   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:21:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:21:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:21:49.225053   16429 conversion.go:110] Could not get instant cpu stats: different number of cpus
>Sep 02 18:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:21:58.678754   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:21:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:22:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:22:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:08.649042   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:22:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:22:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:22:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:22:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Success
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:09.422551   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:09.432898   16429 kuberuntime_container.go:566] Killing container "docker://463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663" with 30 second grace period
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5269]: systemdhook <debug>: 040a3fbca9c6: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5272]: umounthook <debug>: 040a3fbca9c6: only runs in prestart stage, ignoring
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:09.674200   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:09.676440   16429 kuberuntime_container.go:566] Killing container "docker://040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d" with 10 second grace period
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:09.854977802-04:00" level=error msg="Handler for POST /v1.24/containers/040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d/stop?t=10 returned error: Container 040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d is already stopped"
>Sep 02 18:22:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:09.855017039-04:00" level=error msg="Handler for POST /v1.24/containers/040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d/stop returned error: Container 040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d is already stopped"
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[5294]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 veth2fd639d7
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth2fd639d7 left promiscuous mode
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:22:10.072366   16429 docker_sandbox.go:337] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "fod-postgresql-test-1-deploy_test": Unexpected command output nsenter: cannot open /proc/4625/ns/net: No such file or directory
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: with error: exit status 1
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:10.084717   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"875ab363-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d"}
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5314]: systemdhook <debug>: 784f92f75c3d: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5316]: umounthook <debug>: 784f92f75c3d: only runs in prestart stage, ignoring
>Sep 02 18:22:10 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:14.201598492-04:00" level=error msg="Handler for POST /v1.24/containers/784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23/stop?t=10 returned error: Container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23 is already stopped"
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:14.201632792-04:00" level=error msg="Handler for POST /v1.24/containers/784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23/stop returned error: Container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23 is already stopped"
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:14.207463   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"875ab363-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23"}
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:22:14.208001   16429 pod_container_deletor.go:77] Container "784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23" not found in pod's containers
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:14.346331   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "875ab363-aefe-11e8-932d-fa163edaf27e" (UID: "875ab363-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:14.357624   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls" (OuterVolumeSpecName: "deployer-token-85tls") pod "875ab363-aefe-11e8-932d-fa163edaf27e" (UID: "875ab363-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "deployer-token-85tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:22:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:14.447713   16429 reconciler.go:290] Volume detached for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/875ab363-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:22:15 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-besteffort-pod875ab363_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:22:15 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-besteffort-pod875ab363_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:18.015417   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:18.024382   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:18.024905   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-deploy_test(875ab363-aefe-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:18.597337   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.11:5432: getsockopt: connection refused
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:18.659634098-04:00" level=warning msg="containerd: unable to save 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663:efc87c492aa2777b5f07adda24147155d6f6edf7b222067893015c9dce789343 starttime: open /proc/5353/stat: no such file or directory"
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:18.661298308-04:00" level=info msg="containerd: 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663:efc87c492aa2777b5f07adda24147155d6f6edf7b222067893015c9dce789343 (pid 5353) has become an orphan, killing it"
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:18.675759   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:22:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:24.009955   16429 proxier.go:336] Removing service port "test/fod-postgresql-test:5432-tcp"
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: .................................... done
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5417]: systemdhook <debug>: 463bd24bdbfd: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5418]: umounthook <debug>: 463bd24bdbfd: only runs in prestart stage, ignoring
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[5437]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 veth03551a2b
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth03551a2b left promiscuous mode
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5457]: systemdhook <debug>: cef679970fe6: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5458]: umounthook <debug>: cef679970fe6: only runs in prestart stage, ignoring
>Sep 02 18:22:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.318537   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"8f105a78-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663"}
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.319345   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"8f105a78-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a"}
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:22:25.319920   16429 pod_container_deletor.go:77] Container "cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a" not found in pod's containers
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.329138464-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop?t=10 returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.329887337-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.729487   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "8f105a78-aefe-11e8-932d-fa163edaf27e" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.729569   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "fod-postgresql-test" (UniqueName: "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") pod "8f105a78-aefe-11e8-932d-fa163edaf27e" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.743559   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq" (OuterVolumeSpecName: "default-token-brthq") pod "8f105a78-aefe-11e8-932d-fa163edaf27e" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "default-token-brthq". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.755674   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (OuterVolumeSpecName: "fod-postgresql-test") pod "8f105a78-aefe-11e8-932d-fa163edaf27e" (UID: "8f105a78-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e". PluginName "kubernetes.io/glusterfs", VolumeGidValue "2002"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.829838   16429 reconciler.go:290] Volume detached for volume "pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/8f105a78-aefe-11e8-932d-fa163edaf27e-pvc-88c0c43f-aefd-11e8-932d-fa163edaf27e") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.830657   16429 reconciler.go:290] Volume detached for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/8f105a78-aefe-11e8-932d-fa163edaf27e-default-token-brthq") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:25.955927   16429 kubelet_pods.go:996] Killing unwanted pod "fod-postgresql-test-1-bdrdv"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.958144217-04:00" level=error msg="Handler for GET /v1.24/containers/463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663/json returned error: No such container: 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.958932903-04:00" level=error msg="Handler for GET /v1.24/containers/463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663/json returned error: No such container: 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:22:25.963873   16429 remote_runtime.go:277] ContainerStatus "463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663" from runtime service failed: rpc error: code = 2 desc = Error: No such container: 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-burstable-pod8f105a78_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-burstable-pod8f105a78_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.978735189-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop?t=10 returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:25.978771161-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:22:25.981073   16429 kubelet_pods.go:999] Failed killing the pod "fod-postgresql-test-1-bdrdv": failed to "KillContainer" for "postgresql" with KillContainerError: "rpc error: code = 2 desc = Error: No such container: 463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663"
>Sep 02 18:22:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:26.338050230-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop?t=10 returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:22:26.338092352-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:22:26 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:26.379558   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:26 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:26.384427   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:22:26 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:26.393295   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-bdrdv_test(8f105a78-aefe-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:29.660630   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 18:22:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:29.666202   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 18:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:22:34.256815   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506d with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/040a3fbca9c67b569388977ab6a8617d5602808049805865fff0902b3611506dâ: No such file or directory
>Sep 02 18:22:34 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:22:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:37 Starting Node Health Status refresh
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:22:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13987 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:22:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13938 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:22:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:22:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13228 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:22:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:38 Cleaned 0 nodes from health cache
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started DELETE /volumes/7d596038f9459b6b830d9bbf32e36ca8
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:39 Loaded simple allocator
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:22:39 asynchttp.go:125: Started job 6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 64.760457ms
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:39 Started async operation: Delete Volume
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 132.534Âµs
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script snapshot list vol_7d596038f9459b6b830d9bbf32e36ca8 --xml
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: <cliOutput>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opRet>0</opRet>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrno>0</opErrno>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrstr/>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <snapList>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:     <count>0</count>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   </snapList>
>Sep 02 18:22:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: </cliOutput>
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_5226dc43f96b03f119f87e30d82a08fc:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_840194e50899a3fddc6bffc139bb5ec4:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_5226dc43f96b03f119f87e30d82a08fc:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_840194e50899a3fddc6bffc139bb5ec4:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_0bd5c65b8032ce9c02774ccf03281fb0:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_b5c42c69ae37f5cecb60f313a48b434a:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_0bd5c65b8032ce9c02774ccf03281fb0:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_b5c42c69ae37f5cecb60f313a48b434a:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_a22b4d2ac8598977eaed26a42a666ee1:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_c83a16d499706bb9aa97ca6606194e62:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_a22b4d2ac8598977eaed26a42a666ee1:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_c83a16d499706bb9aa97ca6606194e62:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_4ef68640ba06fbe39c3dd46c0f553791:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_a3d3788925bf32b366907e9e6bb58a59:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_4ef68640ba06fbe39c3dd46c0f553791:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_a3d3788925bf32b366907e9e6bb58a59:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_940631515b23e8069e59c40e2558d916:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_940631515b23e8069e59c40e2558d916:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_0fa53e085a568e3cf52242b0f3d97a7f:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_6ab37e414322b2d291b96c5e19d9974c:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_b60f34f85228ad2483c2b3cd65df62fb:
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_0fa53e085a568e3cf52242b0f3d97a7f:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_6ab37e414322b2d291b96c5e19d9974c:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_b60f34f85228ad2483c2b3cd65df62fb:1
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 169.038Âµs
>Sep 02 18:22:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 160.922Âµs
>Sep 02 18:22:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 118.717Âµs
>Sep 02 18:22:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:22:43.015302   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663 with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/463bd24bdbfd2a1efa30582f0dcc765e65ad9ca67faae247471bde9aff252663â: No such file or directory
>Sep 02 18:22:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:22:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 113.536Âµs
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 144.763Âµs
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume stop vol_7d596038f9459b6b830d9bbf32e36ca8 force
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume stop: vol_7d596038f9459b6b830d9bbf32e36ca8: success
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume delete vol_7d596038f9459b6b830d9bbf32e36ca8
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume delete: vol_7d596038f9459b6b830d9bbf32e36ca8: success
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:44 Deleting brick 0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:44 Deleting brick 4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:22:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:44 Deleting brick 5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: umount /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: umount /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.291319ms
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: umount /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvremove -f vg_cca670ded0d3d9406bfe8b42c7d40246/tp_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_4ef68640ba06fbe39c3dd46c0f553791" successfully removed
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_4ef68640ba06fbe39c3dd46c0f553791" successfully removed
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvremove -f vg_c8714ec5934f2236d1e8df8af51d1233/tp_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_0fa53e085a568e3cf52242b0f3d97a7f" successfully removed
>Sep 02 18:22:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_0fa53e085a568e3cf52242b0f3d97a7f" successfully removed
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: rmdir /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_4ef68640ba06fbe39c3dd46c0f553791
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvremove -f vg_5010d0805890e4e816974cdc078d00d9/tp_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_5226dc43f96b03f119f87e30d82a08fc" successfully removed
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_5226dc43f96b03f119f87e30d82a08fc" successfully removed
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 176.99Âµs
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: rmdir /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_0fa53e085a568e3cf52242b0f3d97a7f
>Sep 02 18:22:46 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: sed -i.save "/brick_4ef68640ba06fbe39c3dd46c0f553791/d" /var/lib/heketi/fstab
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: rmdir /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_5226dc43f96b03f119f87e30d82a08fc
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 113.216Âµs
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: sed -i.save "/brick_0fa53e085a568e3cf52242b0f3d97a7f/d" /var/lib/heketi/fstab
>Sep 02 18:22:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: sed -i.save "/brick_5226dc43f96b03f119f87e30d82a08fc/d" /var/lib/heketi/fstab
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:48 Delete Volume succeeded
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:22:48 asynchttp.go:129: Completed job 6872c8a2295057c530fe91fff951de25 in 8.609133758s
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/6872c8a2295057c530fe91fff951de25
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 204 No Content in 161.857Âµs
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:48.498736   16429 proxier.go:336] Removing service port "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:22:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:22:48.809290   16429 roundrobin.go:338] LoadBalancerRR: Removing endpoints for test/glusterfs-dynamic-fod-postgresql-test:
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started DELETE /volumes/203eadac756a761e51c2c2b738bbdddb
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:57 Loaded simple allocator
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 198.702023ms
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:22:57 asynchttp.go:125: Started job 78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:22:57 Started async operation: Delete Volume
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:22:57 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 230.015Âµs
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script snapshot list vol_203eadac756a761e51c2c2b738bbdddb --xml
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: <cliOutput>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opRet>0</opRet>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrno>0</opErrno>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrstr/>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <snapList>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:     <count>0</count>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   </snapList>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: </cliOutput>
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_a22b4d2ac8598977eaed26a42a666ee1:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_c83a16d499706bb9aa97ca6606194e62:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_a22b4d2ac8598977eaed26a42a666ee1:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_c83a16d499706bb9aa97ca6606194e62:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_a3d3788925bf32b366907e9e6bb58a59:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_a3d3788925bf32b366907e9e6bb58a59:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_940631515b23e8069e59c40e2558d916:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_940631515b23e8069e59c40e2558d916:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_6ab37e414322b2d291b96c5e19d9974c:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_b60f34f85228ad2483c2b3cd65df62fb:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_6ab37e414322b2d291b96c5e19d9974c:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_b60f34f85228ad2483c2b3cd65df62fb:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:22:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_840194e50899a3fddc6bffc139bb5ec4:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_840194e50899a3fddc6bffc139bb5ec4:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_0bd5c65b8032ce9c02774ccf03281fb0:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_b5c42c69ae37f5cecb60f313a48b434a:
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_0bd5c65b8032ce9c02774ccf03281fb0:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_b5c42c69ae37f5cecb60f313a48b434a:1
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:22:58 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 120.512Âµs
>Sep 02 18:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:22:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 110.702Âµs
>Sep 02 18:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 155.631Âµs
>Sep 02 18:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume stop vol_203eadac756a761e51c2c2b738bbdddb force
>Sep 02 18:23:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume stop: vol_203eadac756a761e51c2c2b738bbdddb: success
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume delete vol_203eadac756a761e51c2c2b738bbdddb
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume delete: vol_203eadac756a761e51c2c2b738bbdddb: success
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:01 Deleting brick a22b4d2ac8598977eaed26a42a666ee1
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:01 Deleting brick b5c42c69ae37f5cecb60f313a48b434a
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:01 Deleting brick b60f34f85228ad2483c2b3cd65df62fb
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: umount /var/lib/heketi/mounts/vg_e48a8d6ed7013e232c0fecce28594ba4/brick_b5c42c69ae37f5cecb60f313a48b434a
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: umount /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_a22b4d2ac8598977eaed26a42a666ee1
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: umount /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_b60f34f85228ad2483c2b3cd65df62fb
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvremove -f vg_5e9722cc9c43680c03e01726c3db0fb3/tp_a22b4d2ac8598977eaed26a42a666ee1
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_a22b4d2ac8598977eaed26a42a666ee1" successfully removed
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_a22b4d2ac8598977eaed26a42a666ee1" successfully removed
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 120.695Âµs
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: rmdir /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_a22b4d2ac8598977eaed26a42a666ee1
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvremove -f vg_c8714ec5934f2236d1e8df8af51d1233/tp_b60f34f85228ad2483c2b3cd65df62fb
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_b60f34f85228ad2483c2b3cd65df62fb" successfully removed
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_b60f34f85228ad2483c2b3cd65df62fb" successfully removed
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: sed -i.save "/brick_a22b4d2ac8598977eaed26a42a666ee1/d" /var/lib/heketi/fstab
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 101.528Âµs
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: rmdir /var/lib/heketi/mounts/vg_c8714ec5934f2236d1e8df8af51d1233/brick_b60f34f85228ad2483c2b3cd65df62fb
>Sep 02 18:23:02 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:03 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 116.931Âµs
>Sep 02 18:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 138.203Âµs
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvremove -f vg_e48a8d6ed7013e232c0fecce28594ba4/tp_b5c42c69ae37f5cecb60f313a48b434a
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_b5c42c69ae37f5cecb60f313a48b434a" successfully removed
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_b5c42c69ae37f5cecb60f313a48b434a" successfully removed
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: rmdir /var/lib/heketi/mounts/vg_e48a8d6ed7013e232c0fecce28594ba4/brick_b5c42c69ae37f5cecb60f313a48b434a
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: sed -i.save "/brick_b60f34f85228ad2483c2b3cd65df62fb/d" /var/lib/heketi/fstab
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: sed -i.save "/brick_b5c42c69ae37f5cecb60f313a48b434a/d" /var/lib/heketi/fstab
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:05 Delete Volume succeeded
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:23:05 asynchttp.go:129: Completed job 78f5222efc67aa2a725d93ae470529c4 in 8.043212167s
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/78f5222efc67aa2a725d93ae470529c4
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 204 No Content in 120.343Âµs
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:05.759924   16429 proxier.go:336] Removing service port "test/glusterfs-dynamic-data:"
>Sep 02 18:23:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:05.770835   16429 roundrobin.go:338] LoadBalancerRR: Removing endpoints for test/glusterfs-dynamic-data:
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:23:06.561519554-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop?t=10 returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:23:06.561544899-04:00" level=error msg="Handler for POST /v1.24/containers/cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a/stop returned error: Container cef679970fe6e75c0aec3a681a4ed561a96dfb3566868e0b0cfe323e909ae87a is already stopped"
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:23:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:23:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:23:08.279730545-04:00" level=error msg="Handler for POST /v1.24/containers/784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23/stop?t=10 returned error: Container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23 is already stopped"
>Sep 02 18:23:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:23:08.280601636-04:00" level=error msg="Handler for POST /v1.24/containers/784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23/stop returned error: Container 784f92f75c3ddc64fa33d1e3754c8f564ab176161be5051774147240ed12de23 is already stopped"
>Sep 02 18:23:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:09.513645   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started POST /volumes
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 Loaded simple allocator
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 brick_num: 0
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:21.603879   16429 proxier.go:320] Adding new service port "test/fod-postgresql-test:5432-tcp" at 172.30.41.47:5432/TCP
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:21.633558   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 50.858997ms
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:23:21 asynchttp.go:125: Started job 93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 Started async operation: Create Volume
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 Creating brick 2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 Creating brick 1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:21 Creating brick 1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 128.788Âµs
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-besteffort-podcc7b5ab8_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-besteffort-podcc7b5ab8_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:21.712982   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "cc7b5ab8-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:21.814137   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "cc7b5ab8-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/cc7b5ab8-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/cc7b5ab8-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:21.881565   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "cc7b5ab8-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:23:21.883047   16429 container.go:367] Failed to get RecentStats("/system.slice/run-5620.scope") while determining the next housekeeping: unable to find data for container /system.slice/run-5620.scope
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir -p /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir -p /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir -p /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:23:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:22.023509   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 253.932Âµs
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_cca670ded0d3d9406bfe8b42c7d40246/tp_2b8c77aebc532e2519039cf863d5170f -V 1048576K -n brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_2b8c77aebc532e2519039cf863d5170f" created.
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_5010d0805890e4e816974cdc078d00d9/tp_1ec098f25db4f3dd29cee94e1a576434 -V 1048576K -n brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_1ec098f25db4f3dd29cee94e1a576434" created.
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_b8f647c613fc471688feed81d344fffd/tp_1deb01756cabf55fba6fef68e46a9602 -V 1048576K -n brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:23:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_1deb01756cabf55fba6fef68e46a9602" created.
>Sep 02 18:23:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:23 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 123.408Âµs
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_2b8c77aebc532e2519039cf863d5170f isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_1deb01756cabf55fba6fef68e46a9602 isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: awk "BEGIN {print \"/dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_1deb01756cabf55fba6fef68e46a9602 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: awk "BEGIN {print \"/dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_2b8c77aebc532e2519039cf863d5170f /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_1ec098f25db4f3dd29cee94e1a576434 isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 150.437Âµs
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: awk "BEGIN {print \"/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_1ec098f25db4f3dd29cee94e1a576434 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:23:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_cca670ded0d3d9406bfe8b42c7d40246-brick_2b8c77aebc532e2519039cf863d5170f /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_1deb01756cabf55fba6fef68e46a9602 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chown :2001 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_1ec098f25db4f3dd29cee94e1a576434 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chmod 2775 /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 165.343Âµs
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chown :2001 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chown :2001 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chmod 2775 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434/brick
>Sep 02 18:23:25 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chmod 2775 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602/brick
>Sep 02 18:23:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:23:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:23:26 Creating volume vol_80c0e8bf342bda3b708cf4a64ccb97d4 replica 3
>Sep 02 18:23:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:26 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 139.319Âµs
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 112.193Âµs
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d.
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d.
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5699]: systemdhook <debug>: 49fb545541cf: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:23:27 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5700]: umounthook <debug>: prestart container_id:49fb545541cf rootfs:/var/lib/docker/devicemapper/mnt/7740a2ced3a548921e47c787e66239921a1c77306da3dc00f53f144688c9f54f/rootfs
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume create vol_80c0e8bf342bda3b708cf4a64ccb97d4 replica 3 10.10.94.22:/var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f/brick 10.10.94.28:/var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434/brick 10.10.94.39:/var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602/brick
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume create: vol_80c0e8bf342bda3b708cf4a64ccb97d4: success: please start the volume to access data
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:28.244459   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"cc7b5ab8-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d"}
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927008.2944] manager: (veth211ea961): new Veth device (/org/freedesktop/NetworkManager/Devices/21)
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927008.3114] device (veth211ea961): carrier: link connected
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[5725]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth211ea961
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927008.3266] device (veth211ea961): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth211ea961 entered promiscuous mode
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 5.163903ms
>Sep 02 18:23:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:23:29.533756   16429 summary.go:70] Partial failure issuing GetContainerInfoV2: partial failures: ["/system.slice/run-5620.scope": RecentStats: unable to find data for container /system.slice/run-5620.scope]
>Sep 02 18:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 94.895Âµs
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 109.315Âµs
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:23:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51.
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51.
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[5806]: systemdhook <debug>: 09bf538b3b30: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[5807]: umounthook <debug>: prestart container_id:09bf538b3b30 rootfs:/var/lib/docker/devicemapper/mnt/3098e87e20a74e3c6021d9ef65fac010e8a20f5988d98e08230a5d3d5ca5db69/rootfs
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:23:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume start vol_80c0e8bf342bda3b708cf4a64ccb97d4
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume start: vol_80c0e8bf342bda3b708cf4a64ccb97d4: success
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 160.55Âµs
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:31.750709   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"cc7b5ab8-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51"}
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:23:31 Create Volume succeeded
>Sep 02 18:23:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:23:31 asynchttp.go:129: Completed job 93193ce1b70929aa39e8f117df0b6efd in 10.192348599s
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/93193ce1b70929aa39e8f117df0b6efd
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 303 See Other in 157.01Âµs
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /volumes/80c0e8bf342bda3b708cf4a64ccb97d4
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 5.918832ms
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /clusters/d247fd8083c3eeba705dcaf3af33fcca
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 193.935Âµs
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/88667d40e7f24c804d9bdb941400045c
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.361955ms
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/b8388f778784f9d9dde7941d916af81b
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 811.429Âµs
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/cec8b27ee42e56f5ed75b25460c01cdf
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 897.406Âµs
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:32.787900   16429 roundrobin.go:276] LoadBalancerRR: Setting endpoints for test/glusterfs-dynamic-fod-postgresql-test: to [10.10.94.22:1 10.10.94.28:1 10.10.94.39:1]
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:32.788569   16429 roundrobin.go:240] Delete endpoint 10.10.94.22:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:32.788995   16429 roundrobin.go:240] Delete endpoint 10.10.94.28:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:32.789658   16429 roundrobin.go:240] Delete endpoint 10.10.94.39:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:23:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:32.900399   16429 proxier.go:320] Adding new service port "test/glusterfs-dynamic-fod-postgresql-test:" at 172.30.131.25:1/TCP
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Scaling fod-postgresql-test-1 to 1
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:36.881715   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-burstable-podd591bd35_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-burstable-podd591bd35_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:36.988933   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:36.989463   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.093169   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:d3233c7e-aefe-11e8-932d-fa163edaf27e,ResourceVersion:501565,Generation:0,CreationTimestamp:2018-09-02 18:23:32 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.095773   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.096269   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d591bd35-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e.
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d591bd35-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e.
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d591bd35-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/d591bd35-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.144126   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.210333   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:d3233c7e-aefe-11e8-932d-fa163edaf27e,ResourceVersion:501565,Generation:0,CreationTimestamp:2018-09-02 18:23:32 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.373556   16429 glusterfs.go:341] glusterfs: successfully mounted /var/lib/origin/openshift.local.volumes/pods/d591bd35-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.373641   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.386031   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:d3233c7e-aefe-11e8-932d-fa163edaf27e,ResourceVersion:501565,Generation:0,CreationTimestamp:2018-09-02 18:23:32 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.386154   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.386279   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-sw9lh" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:37.517179   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:23:37 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:23:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:23:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:23:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:23:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:23:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b.
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b.
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6066]: systemdhook <debug>: 4b18521397ca: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6067]: umounthook <debug>: prestart container_id:4b18521397ca rootfs:/var/lib/docker/devicemapper/mnt/7f8def3a22a4ec41a89126cb0c2deeefd80d6e755f65af0cf828bfaad012f0bc/rootfs
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:39.237323   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d591bd35-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b"}
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927019.2724] manager: (veth87a36e11): new Veth device (/org/freedesktop/NetworkManager/Devices/22)
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927019.2893] device (veth87a36e11): carrier: link connected
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[6091]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 veth87a36e11
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth87a36e11 entered promiscuous mode
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927019.3178] device (veth87a36e11): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:23:39 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:23:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:23:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:23:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:23:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:23:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container e2c09c61930fa83c1b3a99b2852d336bb3de3a5445e1aa388463e3094c83c237.
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container e2c09c61930fa83c1b3a99b2852d336bb3de3a5445e1aa388463e3094c83c237.
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6171]: systemdhook <debug>: e2c09c61930f: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6172]: umounthook <debug>: prestart container_id:e2c09c61930f rootfs:/var/lib/docker/devicemapper/mnt/6b3fed4db5f236bb4c97d37b6691ef2273b37839950cf8cf05044c9ef54e7f81/rootfs
>Sep 02 18:23:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:41.343674   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d591bd35-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"e2c09c61930fa83c1b3a99b2852d336bb3de3a5445e1aa388463e3094c83c237"}
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The files belonging to this database system will be owned by user "postgres".
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: This user must also own the server process.
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The database cluster will be initialized with locale "en_US.utf8".
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default database encoding has accordingly been set to "UTF8".
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default text search configuration will be set to "english".
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Data page checksums are disabled.
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:23:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating subdirectories ... ok
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:23:43.605Z] [INFO] AnsibleBroker::Catalog
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 10.129.0.1 - - [02/Sep/2018:22:23:43 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 27303
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default max_connections ... 100
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default shared_buffers ... 128MB
>Sep 02 18:23:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting dynamic shared memory implementation ... posix
>Sep 02 18:23:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:46.982269   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:23:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:23:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:23:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:23:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:23:47 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating configuration files ... ok
>Sep 02 18:23:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:56.949587   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:23:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:23:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:23:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:23:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:23:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:23:58.088816   16429 proxier.go:336] Removing service port "test/fod-postgresql-test:5432-tcp"
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started DELETE /volumes/80c0e8bf342bda3b708cf4a64ccb97d4
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:04 Loaded simple allocator
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 94.007995ms
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:24:04 asynchttp.go:125: Started job 39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:04 Started async operation: Delete Volume
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 127.244Âµs
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script snapshot list vol_80c0e8bf342bda3b708cf4a64ccb97d4 --xml
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: <cliOutput>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opRet>0</opRet>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrno>0</opErrno>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <opErrstr/>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   <snapList>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:     <count>0</count>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   </snapList>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: </cliOutput>
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_c83a16d499706bb9aa97ca6606194e62:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_c83a16d499706bb9aa97ca6606194e62:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_2b8c77aebc532e2519039cf863d5170f:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_a3d3788925bf32b366907e9e6bb58a59:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_2b8c77aebc532e2519039cf863d5170f:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_a3d3788925bf32b366907e9e6bb58a59:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_1ec098f25db4f3dd29cee94e1a576434:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_840194e50899a3fddc6bffc139bb5ec4:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_1ec098f25db4f3dd29cee94e1a576434:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_840194e50899a3fddc6bffc139bb5ec4:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_0bd5c65b8032ce9c02774ccf03281fb0:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_0bd5c65b8032ce9c02774ccf03281fb0:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvs --options=lv_name,thin_count --separator=:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   LV:#Thins
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   docker-pool:0
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_1deb01756cabf55fba6fef68e46a9602:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_940631515b23e8069e59c40e2558d916:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_1deb01756cabf55fba6fef68e46a9602:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_940631515b23e8069e59c40e2558d916:1
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   brick_6ab37e414322b2d291b96c5e19d9974c:
>Sep 02 18:24:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   tp_6ab37e414322b2d291b96c5e19d9974c:1
>Sep 02 18:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:05 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 108.258Âµs
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating template1 database in /var/lib/pgsql/data/userdata/base/1 ... ok
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 153.15Âµs
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing pg_authid ... ok
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:24:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:07.027886   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: PANIC:  could not open control file "global/pg_control": Transport endpoint is not connected
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: sh: line 1:    43 Aborted                 "/opt/rh/rh-postgresql95/root/usr/bin/postgres" --single -F -O -c search_path=pg_catalog -c exit_on_error=true template1 > /dev/null
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: child process exited with exit code 134
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initdb: removing contents of data directory "/var/lib/pgsql/data/userdata"
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: could not open directory "/var/lib/pgsql/data/userdata": Transport endpoint is not connected
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initdb: failed to remove contents of data directory
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6323]: systemdhook <debug>: e2c09c61930f: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6324]: umounthook <debug>: e2c09c61930f: only runs in prestart stage, ignoring
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing dependencies ... 
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 632.462Âµs
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:07.695642   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d591bd35-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"e2c09c61930fa83c1b3a99b2852d336bb3de3a5445e1aa388463e3094c83c237"}
>Sep 02 18:24:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:07.956513   16429 desired_state_of_world_populator.go:272] Error processing volume "fod-postgresql-test" for pod "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)": error processing PVC "test"/"fod-postgresql-test": failed to fetch PVC test/fod-postgresql-test from API server. err=persistentvolumeclaims "fod-postgresql-test" not found
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:08.009376   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI} cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI}] Requests:map[memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI} cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:08.013059   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume stop vol_80c0e8bf342bda3b708cf4a64ccb97d4 force
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume stop: vol_80c0e8bf342bda3b708cf4a64ccb97d4: success
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 119.872Âµs
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Success
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:08.422403   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6388]: systemdhook <debug>: 09bf538b3b30: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6389]: umounthook <debug>: 09bf538b3b30: only runs in prestart stage, ignoring
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: gluster --mode=script volume delete vol_80c0e8bf342bda3b708cf4a64ccb97d4
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume delete: vol_80c0e8bf342bda3b708cf4a64ccb97d4: success
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:08 Deleting brick 1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:08 Deleting brick 1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:08 Deleting brick 2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:08.760079   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:08.761453   16429 kuberuntime_container.go:566] Killing container "docker://09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51" with 10 second grace period
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:09.130671849-04:00" level=error msg="Handler for POST /v1.24/containers/09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51/stop?t=10 returned error: Container 09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51 is already stopped"
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:09.136337709-04:00" level=error msg="Handler for POST /v1.24/containers/09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51/stop returned error: Container 09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51 is already stopped"
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.156061   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "fod-postgresql-test" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.157194   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.174862   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq" (OuterVolumeSpecName: "default-token-brthq") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "default-token-brthq". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.187217   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (OuterVolumeSpecName: "fod-postgresql-test") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e". PluginName "kubernetes.io/glusterfs", VolumeGidValue "2001"
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.257969   16429 reconciler.go:290] Volume detached for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:09.258797   16429 reconciler.go:290] Volume detached for volume "pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/d591bd35-aefe-11e8-932d-fa163edaf27e-pvc-cc6c9a18-aefe-11e8-932d-fa163edaf27e") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 115.182Âµs
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[6432]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 veth211ea961
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth211ea961 left promiscuous mode
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: umount /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: umount /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6462]: systemdhook <debug>: 49fb545541cf: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6463]: umounthook <debug>: 49fb545541cf: only runs in prestart stage, ignoring
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: umount /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:24:09 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvremove -f vg_cca670ded0d3d9406bfe8b42c7d40246/tp_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_2b8c77aebc532e2519039cf863d5170f" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_2b8c77aebc532e2519039cf863d5170f" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvremove -f vg_5010d0805890e4e816974cdc078d00d9/tp_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_1ec098f25db4f3dd29cee94e1a576434" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_1ec098f25db4f3dd29cee94e1a576434" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: rmdir /var/lib/heketi/mounts/vg_cca670ded0d3d9406bfe8b42c7d40246/brick_2b8c77aebc532e2519039cf863d5170f
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:24:10.230288   16429 docker_sandbox.go:337] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "fod-postgresql-test-1-deploy_test": Unexpected command output nsenter: cannot open /proc/5686/ns/net: No such file or directory
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: with error: exit status 1
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: rmdir /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_1ec098f25db4f3dd29cee94e1a576434
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 331923f63834196ec62e687426816f1a477bea3eccd4ad32b4530d2ece42a168.
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 331923f63834196ec62e687426816f1a477bea3eccd4ad32b4530d2ece42a168.
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6495]: systemdhook <debug>: 331923f63834: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6496]: umounthook <debug>: prestart container_id:331923f63834 rootfs:/var/lib/docker/devicemapper/mnt/8ec79bd09b350a96ff3c666dde72ec942c46a1c6c25f88e898ea7ee417e0c0e4/rootfs
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: sed -i.save "/brick_2b8c77aebc532e2519039cf863d5170f/d" /var/lib/heketi/fstab
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 112.461Âµs
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: sed -i.save "/brick_1ec098f25db4f3dd29cee94e1a576434/d" /var/lib/heketi/fstab
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvremove -f vg_b8f647c613fc471688feed81d344fffd/tp_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Logical volume "brick_1deb01756cabf55fba6fef68e46a9602" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "tp_1deb01756cabf55fba6fef68e46a9602" successfully removed
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6523]: systemdhook <debug>: 331923f63834: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:24:10 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6524]: umounthook <debug>: 331923f63834: only runs in prestart stage, ignoring
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: rmdir /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_1deb01756cabf55fba6fef68e46a9602
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:11.285993   16429 kubelet_pods.go:996] Killing unwanted pod "fod-postgresql-test-1-sw9lh"
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:11.343792   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"cc7b5ab8-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"09bf538b3b303e27044b5d9c8095b37b6d074f71e3a32f17ee20ddebd811ac51"}
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-burstable-podd591bd35_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-burstable-podd591bd35_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 153.542Âµs
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:11.428756301-04:00" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b: rpc error: code = 2 desc = no such process"
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6557]: systemdhook <debug>: 4b18521397ca: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6559]: umounthook <debug>: 4b18521397ca: only runs in prestart stage, ignoring
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device veth87a36e11 left promiscuous mode
>Sep 02 18:24:11 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.297828   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d591bd35-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b"}
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:24:12.298778   16429 pod_container_deletor.go:77] Container "4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b" not found in pod's containers
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.299223   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"d591bd35-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"331923f63834196ec62e687426816f1a477bea3eccd4ad32b4530d2ece42a168"}
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.313123   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"cc7b5ab8-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d"}
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:24:12.315103   16429 pod_container_deletor.go:77] Container "49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d" not found in pod's containers
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:12.319209124-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop?t=10 returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:12.319989692-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 129.846Âµs
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: sed -i.save "/brick_1deb01756cabf55fba6fef68e46a9602/d" /var/lib/heketi/fstab
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.457010   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "cc7b5ab8-aefe-11e8-932d-fa163edaf27e" (UID: "cc7b5ab8-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.467008   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls" (OuterVolumeSpecName: "deployer-token-85tls") pod "cc7b5ab8-aefe-11e8-932d-fa163edaf27e" (UID: "cc7b5ab8-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "deployer-token-85tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:12.557421   16429 reconciler.go:290] Volume detached for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/cc7b5ab8-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:12 Delete Volume succeeded
>Sep 02 18:24:12 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:24:12 asynchttp.go:129: Completed job 39ae560ac04e0b9eed35ae641d8ab40f in 8.441791992s
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:13.332204654-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop?t=10 returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:24:13.332275733-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/39ae560ac04e0b9eed35ae641d8ab40f
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 204 No Content in 136.657Âµs
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:13.433655   16429 proxier.go:336] Removing service port "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:13.450664   16429 roundrobin.go:338] LoadBalancerRR: Removing endpoints for test/glusterfs-dynamic-fod-postgresql-test:
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-besteffort-podcc7b5ab8_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:13 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-besteffort-podcc7b5ab8_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:17.986280   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:24:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:17.987339   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:24:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:17.987813   16429 reconciler.go:407] Reconciler sync states: could not find pod information in desired or actual states or pending operation, update it in both states: &{volumeName:kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq podName:d591bd35-aefe-11e8-932d-fa163edaf27e volumeSpec:0xc420429020 outerVolumeSpecName:default-token-brthq pod:0xc422d29500 pluginIsAttachable:false volumeGidValue: devicePath: reportedInUse:false mounter:0xc421d24a80}
>Sep 02 18:24:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:17.988499   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.007612   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.022746   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.030880   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-sw9lh_test(d591bd35-aefe-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.068399   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.072682   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.073420   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-deploy_test(cc7b5ab8-aefe-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.100720   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.101736   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq" (OuterVolumeSpecName: "default-token-brthq") pod "d591bd35-aefe-11e8-932d-fa163edaf27e" (UID: "d591bd35-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "default-token-brthq". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:24:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:18.201702   16429 reconciler.go:290] Volume detached for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/d591bd35-aefe-11e8-932d-fa163edaf27e-default-token-brthq") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:24:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:19.965613   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:21.970271   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:23.970932   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:25.971074   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:27.962754   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:29.970544   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started POST /volumes
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 Loaded simple allocator
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 brick_num: 0
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:30.634632   16429 proxier.go:320] Adding new service port "test/fod-postgresql-test:5432-tcp" at 172.30.203.220:5432/TCP
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 202 Accepted in 152.126077ms
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:24:30 asynchttp.go:125: Started job ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 Started async operation: Create Volume
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 Creating brick 00cbfabfbab9b47d8dccf5833b681057
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 Creating brick 089a8b91eac38fbedbe452ea9486955f
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:30 Creating brick fcced2f1f728a4cd0d153f18bf3bff69
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir -p /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir -p /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir -p /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.271726ms
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_cdbff754900b1840633e683839069374/tp_00cbfabfbab9b47d8dccf5833b681057 -V 1048576K -n brick_00cbfabfbab9b47d8dccf5833b681057
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_00cbfabfbab9b47d8dccf5833b681057" created.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_5010d0805890e4e816974cdc078d00d9/tp_089a8b91eac38fbedbe452ea9486955f -V 1048576K -n brick_089a8b91eac38fbedbe452ea9486955f
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_089a8b91eac38fbedbe452ea9486955f" created.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:31.828279   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-besteffort-podf6525e60_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-besteffort-podf6525e60_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:31.945279   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "f6525e60-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:31.964122   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 139.136Âµs
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:32.046640   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "f6525e60-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/f6525e60-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/f6525e60-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/deployer-token-85tls.
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:24:32.070997   16429 container.go:367] Failed to get RecentStats("/system.slice/run-6626.scope") while determining the next housekeeping: unable to find data for container /system.slice/run-6626.scope
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:32.080330   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "fod-postgresql-test-1-deploy" (UID: "f6525e60-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:32.164815   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_cdbff754900b1840633e683839069374-brick_00cbfabfbab9b47d8dccf5833b681057
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_cdbff754900b1840633e683839069374-brick_00cbfabfbab9b47d8dccf5833b681057 isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: lvcreate --poolmetadatasize 8192K -c 256K -L 1048576K -T vg_b8f647c613fc471688feed81d344fffd/tp_fcced2f1f728a4cd0d153f18bf3bff69 -V 1048576K -n brick_fcced2f1f728a4cd0d153f18bf3bff69
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Logical volume "brick_fcced2f1f728a4cd0d153f18bf3bff69" created.
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 112.715Âµs
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: awk "BEGIN {print \"/dev/mapper/vg_cdbff754900b1840633e683839069374-brick_00cbfabfbab9b47d8dccf5833b681057 /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Mounting V5 Filesystem
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_089a8b91eac38fbedbe452ea9486955f
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_089a8b91eac38fbedbe452ea9486955f isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Ending clean mount
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: awk "BEGIN {print \"/dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_089a8b91eac38fbedbe452ea9486955f /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_cdbff754900b1840633e683839069374-brick_00cbfabfbab9b47d8dccf5833b681057 /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31.
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31.
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: mkdir /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057/brick
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6704]: systemdhook <debug>: 53eba2aed91b: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6705]: umounthook <debug>: prestart container_id:53eba2aed91b rootfs:/var/lib/docker/devicemapper/mnt/a5a873477f9cdc116bbce4cda4af6ece6caf29b27f858706c80e6a9ac4e6f9c4/rootfs
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chown :2001 /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057/brick
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 93.48Âµs
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:34.100367   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:34.101416   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"f6525e60-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31"}
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: chmod 2775 /var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057/brick
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927074.1456] manager: (vethfdf8e3a8): new Veth device (/org/freedesktop/NetworkManager/Devices/23)
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927074.1632] device (vethfdf8e3a8): carrier: link connected
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[6738]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vethfdf8e3a8
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device vethfdf8e3a8 entered promiscuous mode
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927074.1871] device (vethfdf8e3a8): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_5010d0805890e4e816974cdc078d00d9-brick_089a8b91eac38fbedbe452ea9486955f /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: mkdir /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f/brick
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chown :2001 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f/brick
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: chmod 2775 /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f/brick
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 200.989Âµs
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_fcced2f1f728a4cd0d153f18bf3bff69
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: meta-data=/dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_fcced2f1f728a4cd0d153f18bf3bff69 isize=512    agcount=8, agsize=32768 blks
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   attr=2, projid32bit=1
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       crc=1        finobt=0, sparse=0
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: data     =                       bsize=4096   blocks=262144, imaxpct=25
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sunit=64     swidth=64 blks
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: log      =internal log           bsize=4096   blocks=2560, version=2
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:          =                       sectsz=512   sunit=64 blks, lazy-count=1
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: realtime =none                   extsz=4096   blocks=0, rtextents=0
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: awk "BEGIN {print \"/dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_fcced2f1f728a4cd0d153f18bf3bff69 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Mounting V5 Filesystem
>Sep 02 18:24:35 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Ending clean mount
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 248.624Âµs
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570.
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570.
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[6826]: systemdhook <debug>: 6f248f821e13: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[6827]: umounthook <debug>: prestart container_id:6f248f821e13 rootfs:/var/lib/docker/devicemapper/mnt/5a0bb913be434472b4773a8c437390e5b0dfc7db15599f3e3404bb85c299d611/rootfs
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:36.275046   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:36.286314   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"f6525e60-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570"}
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:24:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.343551ms
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:37 Starting Node Health Status refresh
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:24:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14582 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:24:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14479 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:24:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:24:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:37.971124   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.453121ms
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b8f647c613fc471688feed81d344fffd-brick_fcced2f1f728a4cd0d153f18bf3bff69 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: mkdir /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69/brick
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: --> Scaling fod-postgresql-test-1 to 1
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:38.813322   16429 kubelet.go:1866] SyncLoop (ADD, "api"): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice libcontainer container kubepods-burstable-podfa7c278c_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting libcontainer container kubepods-burstable-podfa7c278c_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chown :2001 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69/brick
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:38.859930   16429 desired_state_of_world_populator.go:272] Error processing volume "fod-postgresql-test" for pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)": error processing PVC "test"/"fod-postgresql-test": PVC test/fod-postgresql-test has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("")
>Sep 02 18:24:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:38.922940   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:39.024162   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/fa7c278c-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/fa7c278c-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~secret/default-token-brthq.
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 110.986Âµs
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:39.052957   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: chmod 2775 /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69/brick
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: 
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:24:39 Creating volume vol_aa59c5e3c5bf8d1b4f060b21978a9da4 replica 3
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13826 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:39 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:39 Cleaned 0 nodes from health cache
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:39.787160   16429 summary.go:70] Partial failure issuing GetContainerInfoV2: partial failures: ["/system.slice/run-6626.scope": RecentStats: unable to find data for container /system.slice/run-6626.scope]
>Sep 02 18:24:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:39.974703   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 152.92Âµs
>Sep 02 18:24:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume create vol_aa59c5e3c5bf8d1b4f060b21978a9da4 replica 3 10.10.94.22:/var/lib/heketi/mounts/vg_cdbff754900b1840633e683839069374/brick_00cbfabfbab9b47d8dccf5833b681057/brick 10.10.94.28:/var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_089a8b91eac38fbedbe452ea9486955f/brick 10.10.94.39:/var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_fcced2f1f728a4cd0d153f18bf3bff69/brick
>Sep 02 18:24:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume create: vol_aa59c5e3c5bf8d1b4f060b21978a9da4: success: please start the volume to access data
>Sep 02 18:24:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 149.707Âµs
>Sep 02 18:24:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:41.981369   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:42 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 165.542Âµs
>Sep 02 18:24:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 133.893Âµs
>Sep 02 18:24:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:43.967021   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 115.975Âµs
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:24:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: gluster --mode=script volume start vol_aa59c5e3c5bf8d1b4f060b21978a9da4
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: volume start: vol_aa59c5e3c5bf8d1b4f060b21978a9da4: success
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:24:44 Create Volume succeeded
>Sep 02 18:24:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [asynchttp] INFO 2018/09/02 22:24:44 asynchttp.go:129: Completed job ff1a0cb78ac119de07a8fd7ef9ab27ce in 14.001939571s
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /queue/ff1a0cb78ac119de07a8fd7ef9ab27ce
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 303 See Other in 147.953Âµs
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /volumes/aa59c5e3c5bf8d1b4f060b21978a9da4
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 5.98233ms
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /clusters/d247fd8083c3eeba705dcaf3af33fcca
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 161.609Âµs
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/88667d40e7f24c804d9bdb941400045c
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 1.228953ms
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/b8388f778784f9d9dde7941d916af81b
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 700.668Âµs
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Started GET /nodes/cec8b27ee42e56f5ed75b25460c01cdf
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [negroni] Completed 200 OK in 923.589Âµs
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:45.167211   16429 roundrobin.go:276] LoadBalancerRR: Setting endpoints for test/glusterfs-dynamic-fod-postgresql-test: to [10.10.94.22:1 10.10.94.28:1 10.10.94.39:1]
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:45.167305   16429 roundrobin.go:240] Delete endpoint 10.10.94.22:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:45.167317   16429 roundrobin.go:240] Delete endpoint 10.10.94.28:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:45.167324   16429 roundrobin.go:240] Delete endpoint 10.10.94.39:1 for service "test/glusterfs-dynamic-fod-postgresql-test:"
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:24:45.243035   16429 proxier.go:320] Adding new service port "test/glusterfs-dynamic-fod-postgresql-test:" at 172.30.110.50:1/TCP
>Sep 02 18:24:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:45.973578   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:47.966562   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:49.973161   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:51.972686   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:54.015490   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:55.965067   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:57.991759   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:24:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:24:59.964251   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:01.972540   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:03.973883   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:05.980199   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:25:06 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:25:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:07.965309   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:25:08.395529763-04:00" level=error msg="Handler for POST /v1.24/containers/4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b/stop?t=10 returned error: Container 4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b is already stopped"
>Sep 02 18:25:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:25:08.396517707-04:00" level=error msg="Handler for POST /v1.24/containers/4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b/stop returned error: Container 4b18521397ca5b0ea32f88f41e980a57457ad96092c885571496ab9d8982e56b is already stopped"
>Sep 02 18:25:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:25:08.554072928-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop?t=10 returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:25:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:25:08.555248130-04:00" level=error msg="Handler for POST /v1.24/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d/stop returned error: Container 49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d is already stopped"
>Sep 02 18:25:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:09.973598   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:10.555988   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/331923f63834196ec62e687426816f1a477bea3eccd4ad32b4530d2ece42a168 with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/331923f63834196ec62e687426816f1a477bea3eccd4ad32b4530d2ece42a168â: No such file or directory
>Sep 02 18:25:10 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:25:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:11.973727   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:13.965172   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:15.997655   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:17.966010   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:19.970785   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:21.987351   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:23.965209   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:25.971469   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:27.973032   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:28.259825   16429 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: <nil>, rootInodeErr: <nil>, extraDiskErr: du command failed on /var/lib/docker/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7d with output stdout: , stderr: du: cannot access â/var/lib/docker/containers/49fb545541cf854ac063d149f57bc02aebb4079d70f2da3833a4a60dc31c7d7dâ: No such file or directory
>Sep 02 18:25:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: - exit status 1
>Sep 02 18:25:30 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:30.005544   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:31.963952   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:33.972426   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:35.972768   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:25:36 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:25:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:37.981516   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:39.965906   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:41.973392   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:44 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:44.003788   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:45.965248   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:47.972826   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:49.964724   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:51.978600   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:53.962916   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:55.973768   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:57.994170   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:25:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:25:59.964595   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:01.971302   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:03.972337   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:05.972416   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:26:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:07.973073   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:09.964744   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:11.970523   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:13.962104   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:15.972854   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:17.965635   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:19.973478   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:21.971800   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:23.964172   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:25.972975   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:27.971795   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:29.975127   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:31.964617   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:33.972343   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:35.975310   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:26:37 Starting Node Health Status refresh
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:26:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14774 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:26:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:26:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:26:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:26:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:26:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:37.980188   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:26:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13999 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:26:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:26:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:26:38 Cleaned 0 nodes from health cache
>Sep 02 18:26:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:39.979401   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:41.840442   16429 kubelet.go:1606] Unable to mount volumes for pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)": timeout expired waiting for volumes to attach/mount for pod "test"/"fod-postgresql-test-1-47hdz". list of unattached/unmounted volumes=[fod-postgresql-test]; skipping pod
>Sep 02 18:26:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:41.841582   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: timeout expired waiting for volumes to attach/mount for pod "test"/"fod-postgresql-test-1-47hdz". list of unattached/unmounted volumes=[fod-postgresql-test]
>Sep 02 18:26:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:41.972148   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:43.963890   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:45.972299   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:48.003470   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:49.964317   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:51.970534   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:53.963389   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.075883   16429 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-f5930612-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.190109   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:fe46faeb-aefe-11e8-932d-fa163edaf27e,ResourceVersion:501829,Generation:0,CreationTimestamp:2018-09-02 18:24:44 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.194003   16429 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-f5930612-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/fa7c278c-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-f5930612-aefe-11e8-932d-fa163edaf27e.
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/fa7c278c-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-f5930612-aefe-11e8-932d-fa163edaf27e.
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.304143   16429 glusterfs.go:151] glusterfs: endpoints &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:glusterfs-dynamic-fod-postgresql-test,GenerateName:,Namespace:test,SelfLink:/api/v1/namespaces/test/endpoints/glusterfs-dynamic-fod-postgresql-test,UID:fe46faeb-aefe-11e8-932d-fa163edaf27e,ResourceVersion:501829,Generation:0,CreationTimestamp:2018-09-02 18:24:44 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{gluster.kubernetes.io/provisioned-for-pvc: fod-postgresql-test,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.10.94.22  <nil> nil} {10.10.94.28  <nil> nil} {10.10.94.39  <nil> nil}] [] [{ 1 TCP}]}],}
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.374039   16429 glusterfs.go:341] glusterfs: successfully mounted /var/lib/origin/openshift.local.volumes/pods/fa7c278c-aefe-11e8-932d-fa163edaf27e/volumes/kubernetes.io~glusterfs/pvc-f5930612-aefe-11e8-932d-fa163edaf27e
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.374117   16429 operation_generator.go:494] MountVolume.SetUp succeeded for volume "pvc-f5930612-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e") pod "fod-postgresql-test-1-47hdz" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:26:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:54.776577   16429 kuberuntime_manager.go:371] No sandbox for pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)" can be found. Need to start a new one
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:26:55 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Mounting V5 Filesystem
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Ending clean mount
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d.
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d.
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[7304]: systemdhook <debug>: ea9cc2d05fa6: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[7305]: umounthook <debug>: prestart container_id:ea9cc2d05fa6 rootfs:/var/lib/docker/devicemapper/mnt/893eedd1945c8c65fe4751a584a6836e8aa21f3ef72a682d82d6b1f4d96556a4/rootfs
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:56.945922   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:56.954491   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d"}
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>Sep 02 18:26:56 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927216.9986] manager: (vethbdab4586): new Veth device (/org/freedesktop/NetworkManager/Devices/24)
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927217.0083] device (vethbdab4586): carrier: link connected
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[7337]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vethbdab4586
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device vethbdab4586 entered promiscuous mode
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com NetworkManager[13504]: <info>  [1535927217.0447] device (vethbdab4586): enslaved to non-master-type device ovs-system; ignoring
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:26:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:57.976858   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e.
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e.
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[7425]: systemdhook <debug>: ea501e4963d5: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:26:58 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[7426]: umounthook <debug>: prestart container_id:ea501e4963d5 rootfs:/var/lib/docker/devicemapper/mnt/243869a3e6b7d1b48cf989533ab8c338d20c37d7a9ae4bf3bacac9b8bfea2124/rootfs
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:26:59.006368   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e"}
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The files belonging to this database system will be owned by user "postgres".
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: This user must also own the server process.
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The database cluster will be initialized with locale "en_US.utf8".
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default database encoding has accordingly been set to "UTF8".
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: The default text search configuration will be set to "english".
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Data page checksums are disabled.
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
>Sep 02 18:26:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:26:59.973697   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:00 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating subdirectories ... ok
>Sep 02 18:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default max_connections ... 100
>Sep 02 18:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting default shared_buffers ... 128MB
>Sep 02 18:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: selecting dynamic shared memory implementation ... posix
>Sep 02 18:27:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:01.965200   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:03.978765   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:04 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating configuration files ... ok
>Sep 02 18:27:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:05.966295   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:27:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:07.998212   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:08.961805   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:09.967821   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:11.413Z] [INFO] Broker configured to refresh specs every 10m0s seconds
>Sep 02 18:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:11.414Z] [INFO] Attempting bootstrap at 2018-09-02 22:27:11.41356617 +0000 UTC
>Sep 02 18:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:11.414Z] [INFO] AnsibleBroker::Bootstrap
>Sep 02 18:27:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:11.974866   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:13.58Z] [INFO] Validating specs...
>Sep 02 18:27:13 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:13.58Z] [NOTICE] All specs passed validation!
>Sep 02 18:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: !!!!!!filter mode is none[2018-09-02T22:27:14.032Z] [INFO] Validating specs...
>Sep 02 18:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:14.034Z] [NOTICE] All specs passed validation!
>Sep 02 18:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:14.048742   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:14 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [2018-09-02T22:27:14.558Z] [NOTICE] Broker successfully bootstrapped
>Sep 02 18:27:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:15.965842   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:17.974703   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:18.082792   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:18.083622   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:18.084176   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:18.084762   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:18.881632   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:19.982380   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:21.979154   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:23.965029   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:25.973259   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating template1 database in /var/lib/pgsql/data/userdata/base/1 ... ok
>Sep 02 18:27:27 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing pg_authid ... ok
>Sep 02 18:27:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:27.974676   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:28.815366   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:28.887505   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:29.969953   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: initializing dependencies ... ok
>Sep 02 18:27:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:31.975717   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:33.989320   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating system views ... ok
>Sep 02 18:27:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:35.966756   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:27:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:27:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:27:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:27:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:38.001308   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:38.814974   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:38.877140   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: loading system objects' descriptions ... ok
>Sep 02 18:27:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:39.972528   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating collations ... ok
>Sep 02 18:27:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:41.975437   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:43.977437   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:44 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating conversions ... ok
>Sep 02 18:27:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating dictionaries ... ok
>Sep 02 18:27:45 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: setting privileges on built-in objects ... ok
>Sep 02 18:27:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:46.020602   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:47.969410   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:48.815051   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:48.815179   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:48.945367   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:49.119629   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:27:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:49.120430   16429 kuberuntime_container.go:566] Killing container "docker://ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e" with 30 second grace period
>Sep 02 18:27:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:49.992651   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:50 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: creating information schema ... ok
>Sep 02 18:27:51 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: loading PL/pgSQL server-side language ... ok
>Sep 02 18:27:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:51.975287   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:54.002850   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:55.972186   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:57.984621   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:27:58.869908   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:27:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: vacuuming database template1 ... ok
>Sep 02 18:27:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:27:59.977023   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:01.977740   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:04 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:04.001611   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:05.969169   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:28:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:07.976526   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:08.947722   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:09.516664   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:28:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:09.997921   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:10 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: copying template1 to template0 ... ok
>Sep 02 18:28:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:11.979041   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:13.971217   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:15.993826   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:17.968930   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:18.894250   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:28:19.123637061-04:00" level=info msg="Container ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e failed to exit within 30 seconds of signal 15 - using the force"
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[7815]: systemdhook <debug>: ea501e4963d5: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[7816]: umounthook <debug>: ea501e4963d5: only runs in prestart stage, ignoring
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: copying template1 to postgres ... 
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:28:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:19.977735   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:20.502172   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"ea501e4963d5c5297200952bc47890142b4a3edc861b6b4e07ec6598eb88972e"}
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513.
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513.
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[7894]: systemdhook <debug>: 031543d2ad2a: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:28:20 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[7895]: umounthook <debug>: prestart container_id:031543d2ad2a rootfs:/var/lib/docker/devicemapper/mnt/84cb023baada25e420faa3da7a9c8f87787dfe9f20ddd77014e02d6cc4da14a2/rootfs
>Sep 02 18:28:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:21.540083   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513"}
>Sep 02 18:28:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:28:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:21.980406   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:28:22 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:28:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:23.992805   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:25.967938   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:27.976914   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:28.901000   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:30 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:30.012600   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:31.982192   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:33.981196   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:35.978040   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:28:37 Starting Node Health Status refresh
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:28:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14774 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:28:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:28:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:28:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:28:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:28:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:37.989814   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:28:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13999 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:28:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:28:38 Cleaned 0 nodes from health cache
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:38.877151   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:39.972301   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:41.972920   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:43.977719   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:45.966074   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:47.973496   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:48.921809   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:49.978671   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:51.981265   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:53.975890   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:55.968744   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:57.975390   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:58.817156   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:28:58.874139   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:28:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:28:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:28:59.967834   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:01.978613   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:03.964678   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:05.982360   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:29:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:08.016792   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:08.814873   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:08.815770   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:29:08.881680439-04:00" level=warning msg="containerd: unable to save 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513:301722fec9f6592596e6d1e7a49fb07df89e7784d23c481debe5a7556963233b starttime: open /proc/8162/stat: no such file or directory"
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:29:08.882783056-04:00" level=info msg="containerd: 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513:301722fec9f6592596e6d1e7a49fb07df89e7784d23c481debe5a7556963233b (pid 8162) has become an orphan, killing it"
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:08.886306   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:29:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:29:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:09.117405   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:29:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:09.117445   16429 kuberuntime_container.go:566] Killing container "docker://031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513" with 30 second grace period
>Sep 02 18:29:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:09.965419   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:11.974722   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:13.974300   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:15.976385   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:17.966672   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ....................................................... done
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[8217]: systemdhook <debug>: 031543d2ad2a: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[8218]: umounthook <debug>: 031543d2ad2a: only runs in prestart stage, ignoring
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:18.202980   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:18.700151   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513"}
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:29:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:18.990926   16429 remote_runtime.go:309] ExecSync 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513 '/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'' from runtime service failed: rpc error: code = 2 desc = container not running (031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513)
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:19.028736   16429 remote_runtime.go:309] ExecSync 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513 '/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'' from runtime service failed: rpc error: code = 2 desc = container not running (031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513)
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:19.369709   16429 remote_runtime.go:309] ExecSync 031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513 '/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'' from runtime service failed: rpc error: code = 2 desc = container not running (031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513)
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:29:19.370328   16429 prober.go:103] No ref for container "docker://031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513" (fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql)
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:19.370361   16429 prober.go:106] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" errored: rpc error: code = 2 desc = container not running (031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513)
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:29:19 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container 42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30.
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container 42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30.
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[8296]: systemdhook <debug>: 42228a7218ca: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[8297]: umounthook <debug>: prestart container_id:42228a7218ca rootfs:/var/lib/docker/devicemapper/mnt/751b44c5a508d798e82382b134f18f4c2ce6eaaf2326067521a3fc769aded166/rootfs
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:20.274191   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:20.290383   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30"}
>Sep 02 18:29:20 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:29:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:29:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:29:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:21.986015   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:23.968577   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:25.972775   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:27.978397   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:28.901658   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:29:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:29:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:29.964792   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:31.973071   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:33.977703   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:35.972051   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:29:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:37.973729   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:38.873132   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:29:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:29:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:29:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:29:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:29:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:39.964618   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:41.975082   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:43.981010   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:46 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:45.999616   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:47.966703   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:48.883887   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:29:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:29:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:29:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:29:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:29:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:49.975036   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com sshd[8453]: Accepted publickey for quicklab from 10.10.94.219 port 53078 ssh2: RSA SHA256:ml0ifl/iSfmuGb9yMNV2hTopaGUQZbonxq9gEnL3rI0
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd-logind[496]: New session 65 of user quicklab.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Created slice User Slice of quicklab.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting User Slice of quicklab.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started Session 65 of user quicklab.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com sshd[8453]: pam_unix(sshd:session): session opened for user quicklab by (uid=0)
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting Session 65 of user quicklab.
>Sep 02 18:29:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:51.967573   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:53.975929   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:55 node-0.alxcns2.lab.rdu2.cee.redhat.com sudo[8521]: quicklab : TTY=pts/0 ; PWD=/home/quicklab ; USER=root ; COMMAND=/bin/su -
>Sep 02 18:29:55 node-0.alxcns2.lab.rdu2.cee.redhat.com su[8522]: (to root) quicklab on pts/0
>Sep 02 18:29:55 node-0.alxcns2.lab.rdu2.cee.redhat.com su[8522]: pam_unix(su-l:session): session opened for user root by quicklab(uid=0)
>Sep 02 18:29:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:55.966008   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:57.974122   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:58.820998   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:29:58.887972   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:29:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:29:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:29:59.978785   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:01.963047   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:03.974512   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:05.975165   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:30:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:07.976686   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:08.814656   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:08.815687   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:30:08.894604415-04:00" level=warning msg="containerd: unable to save 42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30:359598f7799d8c22540f6494524fca44514889359179d260e4d3eb5cf1a27824 starttime: open /proc/8634/stat: no such file or directory"
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:30:08.898631991-04:00" level=info msg="containerd: 42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30:359598f7799d8c22540f6494524fca44514889359179d260e4d3eb5cf1a27824 (pid 8634) has become an orphan, killing it"
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:08.902052   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:30:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:30:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:09.117249   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:30:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:09.117983   16429 kuberuntime_container.go:566] Killing container "docker://42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30" with 30 second grace period
>Sep 02 18:30:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:09.967598   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:11.978247   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:13.977595   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:15.981095   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:17.981117   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:18.178629   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:18.179628   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:18.180168   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:18.181986   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:18.927529   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:30:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:30:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:19.975501   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ........................................................... stopped waiting
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server is still starting up
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  the database system is starting up
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[8710]: systemdhook <debug>: 42228a7218ca: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[8712]: umounthook <debug>: 42228a7218ca: only runs in prestart stage, ignoring
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:21.570028   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:30:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:21.966003   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:22 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:30:22 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:22.173640   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"42228a7218cabc07182b61d5cbe665162ce362f5aebfa7249a6ea8c63158ff30"}
>Sep 02 18:30:22 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:30:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:30:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:30:24 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:24.667421   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:24 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07.
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07.
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[8791]: systemdhook <debug>: bc25fbfc3281: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:30:25 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[8792]: umounthook <debug>: prestart container_id:bc25fbfc3281 rootfs:/var/lib/docker/devicemapper/mnt/5a75037912ffb44038156af2b93d7c46150e90aae1d23d2f03cb437cf5866062/rootfs
>Sep 02 18:30:26 node-0.alxcns2.lab.rdu2.cee.redhat.com su[8522]: pam_unix(su-l:session): session closed for user root
>Sep 02 18:30:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:27.473930   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:27.493069   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513"}
>Sep 02 18:30:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:30:27.493855   16429 pod_container_deletor.go:77] Container "031543d2ad2a517bc72a45a62c3ec839d3c96e00caf3a817c55a0c7414a4a513" not found in pod's containers
>Sep 02 18:30:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:27.494319   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07"}
>Sep 02 18:30:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:27.981004   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:30:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:29.967481   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:31.982953   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:33.967461   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:35.977677   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:30:37 Starting Node Health Status refresh
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:30:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14774 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:30:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:30:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:30:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:37.973367   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:30:38 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:30:38 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:30:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13999 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:30:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:30:38 Cleaned 0 nodes from health cache
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:38.926594   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:30:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:30:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:39.990164   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:41.967883   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:43.977391   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:45.969320   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:47.976213   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:48.886447   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:30:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:30:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:30:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:30:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:30:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:49.974109   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:52 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:52.023107   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:53.964711   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:55.974629   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:57.988859   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:58.815330   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:30:58.877272   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:30:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:30:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:30:59.977038   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:01.978019   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:03.973791   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:05.974786   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:31:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:07.965559   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:08.814760   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:08.815578   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:08.886565   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:31:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:31:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:09.117043   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:31:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:09.117781   16429 kuberuntime_container.go:566] Killing container "docker://bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07" with 30 second grace period
>Sep 02 18:31:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:09.974673   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:12.023212   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:13.964457   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:15.998365   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:17.968802   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:18.878427   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:31:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:31:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:31:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:31:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:31:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:19.973946   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:21.980389   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:23.967874   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:25.973248   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:27.980293   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ........................................................... stopped waiting
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server is still starting up
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[9142]: systemdhook <debug>: bc25fbfc3281: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[9143]: umounthook <debug>: bc25fbfc3281: only runs in prestart stage, ignoring
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:31:28.941793023-04:00" level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value \"oci runtime error: exec failed: container \\\"bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07\\\" does not exist\\n\""
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:28.944600   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container \"bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07\" does not exist\n"
>Sep 02 18:31:28 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:29.069510   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:29.489443   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"bc25fbfc32818c2f843c5084df0d512ccee9ae49d590ca7a06c9854698e29f07"}
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:31:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:29.966882   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950.
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950.
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[9221]: systemdhook <debug>: b92b9848c90e: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[9222]: umounthook <debug>: prestart container_id:b92b9848c90e rootfs:/var/lib/docker/devicemapper/mnt/cbde669767daac26a2e6ccc86b3b42ba640d670bd11fe4dd3ed93f9e4873546d/rootfs
>Sep 02 18:31:30 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:30.961985   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950"}
>Sep 02 18:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:31:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:31.985582   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:31:32 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:31:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:33.978161   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:35.966452   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:31:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:37.975790   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:38.924992   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:31:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:31:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:39.965163   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:41.975722   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:43.968002   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:45.976611   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:47.974642   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:48.873044   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:31:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:31:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:31:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:31:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:31:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:49.976366   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:51.966528   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:53.974297   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:55.965055   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:57.973606   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:31:58.879088   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:31:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:31:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:31:59.965124   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:01.975820   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:03.973801   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:05.977249   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:32:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:07.982031   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:08.815078   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:08.878347   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:09.965811   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:11.994195   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:13.976296   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:15.967029   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:17.977713   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:18.818392   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:18.819446   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:18.881870   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:19.122499   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:32:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:19.123476   16429 kuberuntime_container.go:566] Killing container "docker://b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950" with 30 second grace period
>Sep 02 18:32:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:19.983114   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:21.965478   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:23.975417   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:25.975811   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:27.967585   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:28.910100   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:29.973668   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: .......................................................... done
>Sep 02 18:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:32:30 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[9569]: systemdhook <debug>: b92b9848c90e: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[9570]: umounthook <debug>: b92b9848c90e: only runs in prestart stage, ignoring
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:31.086109   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:32:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:32.206086   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:32.215162   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"b92b9848c90ef30573c6ad08317ed90fbe8b8f222a46be6d77b316e2991ec950"}
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Mounting V5 Filesystem
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Ending clean mount
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started docker container bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101.
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting docker container bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101.
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[9650]: systemdhook <debug>: bce2b983018c: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:32:32 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[9651]: umounthook <debug>: prestart container_id:bce2b983018c rootfs:/var/lib/docker/devicemapper/mnt/4147ed48a8294ce6a37071a440026f1d404d85868afe81c7ee31309858f9cc7b/rootfs
>Sep 02 18:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:33.268582   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerStarted", Data:"bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101"}
>Sep 02 18:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: pg_ctl: another server might be running; trying to start server anyway
>Sep 02 18:32:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:33.979565   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: waiting for server to start....LOG:  redirecting log output to logging collector process
>Sep 02 18:32:34 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: HINT:  Future log output will appear in directory "pg_log".
>Sep 02 18:32:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:35.966447   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:32:37 Starting Node Health Status refresh
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:32:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14774 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:32:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:32:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:32:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:32:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:32:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:38.007540   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:32:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13999 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:32:38 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:32:38 Cleaned 0 nodes from health cache
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:38.926347   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:38 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:39.966403   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:41.978775   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:43.980331   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:45.967340   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:47.978629   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:48.880789   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:49.979365   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:51.965573   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:54 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:54.002116   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:55.966096   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:57.974195   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:32:58.874958   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:32:58 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:32:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:32:59.967655   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:01.974789   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:03.966344   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:04 node-0.alxcns2.lab.rdu2.cee.redhat.com rhnsd[908]: /etc/sysconfig/rhn/systemid does not exist or is unreadable
>Sep 02 18:33:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:05.976330   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:33:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:07.982146   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:08.815172   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:08.874891   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:33:08 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:09.519699   16429 container_manager_linux.go:399] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
>Sep 02 18:33:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:09.965544   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:11.997360   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:13.973026   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:15.965862   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:17.979219   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:18.207314   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:18.208409   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:18.208945   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:18.209552   16429 reconciler.go:367] Could not construct volume information: impossible to reconstruct glusterfs volume spec from volume mountpath
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:18.814951   16429 prober.go:111] Liveness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): dial tcp 10.128.0.15:5432: getsockopt: connection refused
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:18.815819   16429 kubelet.go:1933] SyncLoop (container unhealthy): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:18.887621   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:33:18 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:33:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:19.117497   16429 kuberuntime_manager.go:525] Container "postgresql" ({"docker" "bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101"}) of pod fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e): Container failed liveness probe.. Container will be killed and recreated.
>Sep 02 18:33:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:19.117571   16429 kuberuntime_container.go:566] Killing container "docker://bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101" with 30 second grace period
>Sep 02 18:33:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:19.965558   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:21.975808   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:24 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:24.005261   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:25.967638   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:27.974403   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:28.878426   16429 prober.go:111] Readiness probe for "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e):postgresql" failed (failure): sh: cannot set terminal process group (-1): Inappropriate ioctl for device
>Sep 02 18:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: sh: no job control in this shell
>Sep 02 18:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: psql: could not connect to server: Connection refused
>Sep 02 18:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: Is the server running on host "127.0.0.1" and accepting
>Sep 02 18:33:28 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: TCP/IP connections on port 5432?
>Sep 02 18:33:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:29.978102   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: ......................................................... done
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: server started
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: psql: FATAL:  database "postgres" does not exist
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[9990]: systemdhook <debug>: bce2b983018c: Skipping as container command is container-entrypoint, not init or systemd
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[9992]: umounthook <debug>: bce2b983018c: only runs in prestart stage, ignoring
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-14): Unmounting Filesystem
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:31.928045   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:31.929042   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:31.929692   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:31.975385   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:32.071221   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"bce2b983018cb4317c1325a6dd8cd26cd0c2305b378fc73b2632c91047fc8101"}
>Sep 02 18:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:32.381941   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:32.386864   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:32.387091   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:33:32 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:32.387184   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:33.965261   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:35.994339   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:33:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:37.973316   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:39.973414   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:41.964318   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:43.972603   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:45.965949   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:47.974906   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:48.257384   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] Requests:map[memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI} cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:33:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:48.260875   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:33:48.261077   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:33:48 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:48.261113   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:33:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:49.965580   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:51.973566   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:53.972715   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:55.965080   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:57.973505   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:33:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:33:59.965120   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:00.256921   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:00.260446   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:00.260658   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:34:00 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:00.260690   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:01.972627   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:03.973003   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:05.967913   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:34:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:07.994161   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:09.965330   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:11 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:11.972084   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:13.965962   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:15.250905   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:15.254788   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:15.255016   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:15.255065   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:15.972202   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:17.963340   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:19.969721   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:21.964071   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:23.972279   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:25.964307   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:27.972331   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:29.251407   16429 kuberuntime_manager.go:489] Container {Name:postgresql Image:registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:1e2e4fba080f6876c8b8099160f6471f2bed939afb97c839d6e20357aa239b03 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:5432 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POSTGRESQL_USER Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-user,Optional:nil,},}} {Name:POSTGRESQL_PASSWORD Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:fod-postgresql-test,},Key:database-password,Optional:nil,},}} {Name:POSTGRESQL_DATABASE Value:foddb ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:1 scale:0} d:{Dec:<nil>} s:1 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:536870912 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:fod-postgresql-test ReadOnly:false MountPath:/var/lib/pgsql/data SubPath:} {Name:default-token-brthq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:5432,Host:,},},InitialDelaySeconds:30,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL MKNOD SETGID SETUID],},Privileged:*false,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c11,c5,},RunAsUser:*1000120000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
>Sep 02 18:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:29.255284   16429 kuberuntime_manager.go:733] checking backoff for container "postgresql" in pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:29.255483   16429 kuberuntime_manager.go:743] Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)
>Sep 02 18:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:29.255515   16429 pod_workers.go:186] Error syncing pod fa7c278c-aefe-11e8-932d-fa163edaf27e ("fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"), skipping: failed to "StartContainer" for "postgresql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=postgresql pod=fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:29.964919   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:31.974119   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:33.980655   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:35 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:34:35.074215       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of *v1.StorageClass ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:34:36 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:36.000812   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:34:37 Starting Node Health Status refresh
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:34:37 Check Glusterd service status in node infra-1.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-1.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-p9pln Command: systemctl status glusterd
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:12 UTC; 2 days ago
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410e5ee_acfd_11e8_932d_fa163edaf27e.slice/docker-aee785afa9e5b40de4bcc14a59d1056a955d3c93971e626d10726f98e85fef06.scope/system.slice/glusterd.service
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  921 /usr/sbin/glusterfsd -s 10.10.94.22 --volfile-id heketidbstorage.10.10.94.22.var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.22-var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.pid -S /var/run/gluster/ccd5c7a602d2dee6ab3d88a59599eacd.socket --brick-name /var/lib/heketi/mounts/vg_5e9722cc9c43680c03e01726c3db0fb3/brick_c83a16d499706bb9aa97ca6606194e62/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5e9722cc9c43680c03e01726c3db0fb3-brick_c83a16d499706bb9aa97ca6606194e62-brick.log --xlator-option *-posix.glusterd-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14774 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/62f85c157fcd0096878c7fad5249e690.socket --xlator-option *replicate*.node-uuid=b9ccb80a-36b7-4e0f-a80f-dfdb5c65d353
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:06 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:12 infra-1.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:34:37 Periodic health check status: node 88667d40e7f24c804d9bdb941400045c up=true
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:34:37 Check Glusterd service status in node infra-0.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-0.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-9qllq Command: systemctl status glusterd
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:19 UTC; 2 days ago
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 409 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 410 (glusterd)
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9b25_acfd_11e8_932d_fa163edaf27e.slice/docker-b7b9bf9f4f80d007fceaac459e547e17468908f73930f37a46cdce118ece391d.scope/system.slice/glusterd.service
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  410 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  897 /usr/sbin/glusterfsd -s 10.10.94.28 --volfile-id heketidbstorage.10.10.94.28.var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.28-var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.pid -S /var/run/gluster/20d4d73740f2207f52615e5eb2ecf92b.socket --brick-name /var/lib/heketi/mounts/vg_5010d0805890e4e816974cdc078d00d9/brick_840194e50899a3fddc6bffc139bb5ec4/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5010d0805890e4e816974cdc078d00d9-brick_840194e50899a3fddc6bffc139bb5ec4-brick.log --xlator-option *-posix.glusterd-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ14639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/e2fd1345e8284114357757142e0529be.socket --xlator-option *replicate*.node-uuid=ce5da9cf-21c5-493f-81a0-d746d2f3255b
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:14 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:19 infra-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:34:37 Periodic health check status: node b8388f778784f9d9dde7941d916af81b up=true
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [cmdexec] INFO 2018/09/02 22:34:37 Check Glusterd service status in node infra-2.alxcns2.lab.rdu2.cee.redhat.com
>Sep 02 18:34:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:37.974787   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: error: update acceptor rejected fod-postgresql-test-1: pods for rc 'test/fod-postgresql-test-1' took longer than 600 seconds to become available
>Sep 02 18:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[10149]: systemdhook <debug>: 6f248f821e13: Skipping as container command is /usr/bin/openshift-deploy, not init or systemd
>Sep 02 18:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[10150]: umounthook <debug>: 6f248f821e13: only runs in prestart stage, ignoring
>Sep 02 18:34:38 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-9): Unmounting Filesystem
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:39.948839   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"f6525e60-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570"}
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:39.955631   16429 kubelet_pods.go:996] Killing unwanted pod "fod-postgresql-test-1-deploy"
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:39.980461   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:39.981466   16429 kuberuntime_container.go:566] Killing container "docker://6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570" with 10 second grace period
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:34:39.983276770-04:00" level=error msg="Handler for POST /v1.24/containers/6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570/stop?t=10 returned error: Container 6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570 is already stopped"
>Sep 02 18:34:39 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:34:39.983322231-04:00" level=error msg="Handler for POST /v1.24/containers/6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570/stop returned error: Container 6f248f821e131e38067f897147052e0b4980594806cddf2b2c4a35512fc07570 is already stopped"
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:40.005934   16429 kuberuntime_container.go:66] Can't make a ref to pod "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)", container deployment: selfLink was empty, can't make reference
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:40.041390   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[10172]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 vethfdf8e3a8
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device vethfdf8e3a8 left promiscuous mode
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:40.105809   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") pod "f6525e60-aefe-11e8-932d-fa163edaf27e" (UID: "f6525e60-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:40.114932   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls" (OuterVolumeSpecName: "deployer-token-85tls") pod "f6525e60-aefe-11e8-932d-fa163edaf27e" (UID: "f6525e60-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "deployer-token-85tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:34:40.156816   16429 docker_sandbox.go:337] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "fod-postgresql-test-1-deploy_test": Unexpected command output nsenter: cannot open /proc/6691/ns/net: No such file or directory
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: with error: exit status 1
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:34:40.159124962-04:00" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31: rpc error: code = 2 desc = no such process"
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[10202]: systemdhook <debug>: 53eba2aed91b: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[10203]: umounthook <debug>: 53eba2aed91b: only runs in prestart stage, ignoring
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:40.206654   16429 reconciler.go:290] Volume detached for volume "deployer-token-85tls" (UniqueName: "kubernetes.io/secret/f6525e60-aefe-11e8-932d-fa163edaf27e-deployer-token-85tls") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-7): Unmounting Filesystem
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:40.969555   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-deploy_test(f6525e60-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"f6525e60-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31"}
>Sep 02 18:34:40 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:34:40.970601   16429 pod_container_deletor.go:77] Container "53eba2aed91bf849fab2242ac4dc6f565f5c740c1086fc1806e006fa10a8aa31" not found in pod's containers
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [kubeexec] DEBUG 2018/09/02 22:34:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: infra-2.alxcns2.lab.rdu2.cee.redhat.com Pod: glusterfs-storage-2nn68 Command: systemctl status glusterd
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Result: â glusterd.service - GlusterFS, a clustered file-system server
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    Active: active (running) since Fri 2018-08-31 09:11:17 UTC; 2 days ago
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:   Process: 420 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:  Main PID: 421 (glusterd)
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:    CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a7e1_acfd_11e8_932d_fa163edaf27e.slice/docker-977a434102beb67a8e5355c07d48bf5f41044ecc8f085e2058b9f76013324356.scope/system.slice/glusterd.service
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  421 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ  879 /usr/sbin/glusterfsd -s 10.10.94.39 --volfile-id heketidbstorage.10.10.94.39.var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick -p /var/run/gluster/vols/heketidbstorage/10.10.94.39-var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.pid -S /var/run/gluster/0f9163c08dd6e99a72ffa27c9fec91c7.socket --brick-name /var/lib/heketi/mounts/vg_b8f647c613fc471688feed81d344fffd/brick_940631515b23e8069e59c40e2558d916/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b8f647c613fc471688feed81d344fffd-brick_940631515b23e8069e59c40e2558d916-brick.log --xlator-option *-posix.glusterd-uuid=aba70564-c62b-4152-904e-b42e883b2ef3 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]:            ââ13999 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a35c15ca2ccb3bb210e1ad28a36a7c86.socket --xlator-option *replicate*.node-uuid=aba70564-c62b-4152-904e-b42e883b2ef3
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: 
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:11 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: Aug 31 09:11:17 infra-2.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:34:41 Periodic health check status: node cec8b27ee42e56f5ed75b25460c01cdf up=true
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: [heketi] INFO 2018/09/02 22:34:41 Cleaned 0 nodes from health cache
>Sep 02 18:34:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:41.964499   16429 kubelet_pods.go:996] Killing unwanted pod "fod-postgresql-test-1-47hdz"
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:42.003784   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-besteffort-podf6525e60_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-besteffort-podf6525e60_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com ovs-vsctl[10231]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 vethbdab4586
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: device vethbdab4586 left promiscuous mode
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.116892   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq") pod "fa7c278c-aefe-11e8-932d-fa163edaf27e" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.117608   16429 reconciler.go:186] operationExecutor.UnmountVolume started for volume "fod-postgresql-test" (UniqueName: "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e") pod "fa7c278c-aefe-11e8-932d-fa163edaf27e" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e")
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.152310   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq" (OuterVolumeSpecName: "default-token-brthq") pod "fa7c278c-aefe-11e8-932d-fa163edaf27e" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "default-token-brthq". PluginName "kubernetes.io/secret", VolumeGidValue ""
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.163658   16429 operation_generator.go:554] UnmountVolume.TearDown succeeded for volume "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e" (OuterVolumeSpecName: "fod-postgresql-test") pod "fa7c278c-aefe-11e8-932d-fa163edaf27e" (UID: "fa7c278c-aefe-11e8-932d-fa163edaf27e"). InnerVolumeSpecName "pvc-f5930612-aefe-11e8-932d-fa163edaf27e". PluginName "kubernetes.io/glusterfs", VolumeGidValue "2001"
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-systemd-hook[10261]: systemdhook <debug>: ea9cc2d05fa6: Skipping as container command is /usr/bin/pod, not init or systemd
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com oci-umount[10263]: umounthook <debug>: ea9cc2d05fa6: only runs in prestart stage, ignoring
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.218338   16429 reconciler.go:290] Volume detached for volume "default-token-brthq" (UniqueName: "kubernetes.io/secret/fa7c278c-aefe-11e8-932d-fa163edaf27e-default-token-brthq") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:42.218369   16429 reconciler.go:290] Volume detached for volume "pvc-f5930612-aefe-11e8-932d-fa163edaf27e" (UniqueName: "kubernetes.io/glusterfs/fa7c278c-aefe-11e8-932d-fa163edaf27e-pvc-f5930612-aefe-11e8-932d-fa163edaf27e") on node "node-0.alxcns2.lab.rdu2.cee.redhat.com" DevicePath ""
>Sep 02 18:34:42 node-0.alxcns2.lab.rdu2.cee.redhat.com kernel: XFS (dm-13): Unmounting Filesystem
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:43.049126   16429 kubelet.go:1900] SyncLoop (PLEG): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", event: &pleg.PodLifecycleEvent{ID:"fa7c278c-aefe-11e8-932d-fa163edaf27e", Type:"ContainerDied", Data:"ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d"}
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: W0902 18:34:43.050680   16429 pod_container_deletor.go:77] Container "ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d" not found in pod's containers
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:34:43.955633   16429 kubelet_pods.go:996] Killing unwanted pod "fod-postgresql-test-1-47hdz"
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:34:43.963393402-04:00" level=error msg="Handler for POST /v1.24/containers/ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d/stop?t=10 returned error: Container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d is already stopped"
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:34:43.964266039-04:00" level=error msg="Handler for POST /v1.24/containers/ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d/stop returned error: Container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d is already stopped"
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:43.969257   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Removed slice libcontainer container kubepods-burstable-podfa7c278c_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:34:43 node-0.alxcns2.lab.rdu2.cee.redhat.com systemd[1]: Stopping libcontainer container kubepods-burstable-podfa7c278c_aefe_11e8_932d_fa163edaf27e.slice.
>Sep 02 18:34:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:45.969984   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:47.963754   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:49.968848   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:51 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:51.971303   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:53 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:53.963597   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:55 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:55.970896   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:57 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:57.964865   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:34:59 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:34:59.972093   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:01 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:01.980078   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:03 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:03.972405   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:05 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:05.973069   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: W0902 22:35:07.737275       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of *v1.PersistentVolumeClaim ended with: etcdserver: mvcc: required revision has been compacted
>Sep 02 18:35:07 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:07.964903   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:35:08.916505529-04:00" level=error msg="Handler for POST /v1.24/containers/ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d/stop?t=10 returned error: Container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d is already stopped"
>Sep 02 18:35:08 node-0.alxcns2.lab.rdu2.cee.redhat.com dockerd-current[12952]: time="2018-09-02T18:35:08.917593371-04:00" level=error msg="Handler for POST /v1.24/containers/ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d/stop returned error: Container ea9cc2d05fa65e02edd3fa3a5fd89c6f79d4fcbc2c1ec84afb88435925910c4d is already stopped"
>Sep 02 18:35:09 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:09.973335   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:12 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:12.005178   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:13 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:13.963373   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:15 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:15.986590   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:17 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:17.981852   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:35:19.660679   16429 prober.go:111] Liveness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 18:35:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:35:19.665614   16429 prober.go:111] Readiness probe for "asb-1-xlmdw_openshift-ansible-service-broker(e3ed67ef-acff-11e8-932d-fa163edaf27e):asb" failed (failure): Get https://10.128.0.5:1338/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
>Sep 02 18:35:19 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:19.963077   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:35:21.324681   16429 kubelet.go:1882] SyncLoop (DELETE, "api"): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:35:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:35:21.333065   16429 kubelet.go:1876] SyncLoop (REMOVE, "api"): "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)"
>Sep 02 18:35:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: I0902 18:35:21.342420   16429 kubelet.go:2067] Failed to delete pod "fod-postgresql-test-1-47hdz_test(fa7c278c-aefe-11e8-932d-fa163edaf27e)", err: pod not found
>Sep 02 18:35:21 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:21.967310   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:23 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:23.970635   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:25 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:25.963679   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:27 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:27.972624   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:29 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:29.963375   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:31 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:31.970374   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:33 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:33.967049   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:35 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:35.972151   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: setting upstream servers from DBus
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.3#53
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 10.11.5.4#53
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com dnsmasq[13685]: using nameserver 127.0.0.1#53 for domain cluster.local
>Sep 02 18:35:37 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:37.972376   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:39 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:39.963295   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:41 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:41.986185   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:43 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:43.962659   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:45 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:45.970728   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:47 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:47.966682   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
>Sep 02 18:35:49 node-0.alxcns2.lab.rdu2.cee.redhat.com atomic-openshift-node[16429]: E0902 18:35:49.970884   16429 kubelet_volumes.go:128] Orphaned pod "d591bd35-aefe-11e8-932d-fa163edaf27e" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.

Actions: View

Attachments on bug 1580024: 1438867 | 1480400 | 1480401